I’m a heavy user and also a code contributor to Apache Superset. Running Superset on my MacBook is the only reason to have a Docker(still a VM inside?) installed which I think is too heavy.

Superset puts most heavy work onto the database side, I was thinking is there may be some possibility to have a Superset.app to make it easier to use Superset on my MacBook.

My technical stack is mainly backend, some keywords like:

Python, Golang, C, Backend

I have absolutely no experience with Objective-C or Swift and don’t even understand how a macOS App and a UNIX executable are related. But it still deserves a try.

Packaging

Apache Superset is a Python + React project, to avoiding introduce a new stack, I think using Python to develop the UI part might be easy. For the backend, everyone using Python knows that the packaging is a really complex problem. Especially with Python packages including dynamically linked shared libraries(*.dylib or *.so).

Some people may say what about using pip freeze to generate a requirement.txt and install them on the first launch. Well, there may be mainly 2 problems:

1. It will break the convention of “Dragging into Trash means Uninstall”.
2. The user’s Python environment may also be polluted and cause various problems.

After some Googling, I found there are mainly 3 tools I can use to do the packaging:

1. PyInstaller (9.2k stars, Windows/Linux/macOS)
2. cx_Freeze (905 stars, Windows/Linux/macOS)
3. Py2app (189 stars, macOS only but actively developing)

PyInstaller is a pretty old project, a lot of Python-based Windows programs are built with it, and not to mention, it has 9.2k stars. Typically the most “stared” tool is the best choice, but not this time.

As far as I know, there is no silver bullet to solve these *.dylib packaging problems. Most of the packaging tools have to keep a bunch of recipes to solve the tricky libs. Superset introduces a lot of database drivers which are hard to deal with. With years of experience solving the *.so problems, I need a relatively simple tool in case of hacking is needed. So my decision is to give py2app a try.

Dynamic Libs

I suffered from the Linux dynamic lineage problems. Linux kernel ABI is quite stable, and Glibc handles the ABI very carefully.

Even libs like OpenSSL tried their best to keep a stable ABI.

But the upper-level libs shipped with different Linux distributions and pkg management systems constructed a matrix.

When it comes to macOS, most libs shipped with the system are quite stable and things in Linux like libc, libc++, libpthread, libm are merged into libSystem.B.dylib.

Although, the ps command in macOS is not GNU version, you can still see the big difference of dependent libs of /bin/ps between CentOS 8.1andmacOS 12.3.

Solving the packaging of dynamic libs is quite nasty. But I think the main principles are just two:

1. Keep necessary lib at its lowest version

2. Trim lib to minimal from the upper level

The first principle is easy to understand, lower version lib usually means less API required. But AFAIK, macOS doesn’t provide any cross-compiling way to do that. All I have to do is compile SuperChart on a lower version of macOS. To do that I brought a second-hand Mac Mini released in 2009 and installed macOS 10.15 with macOS Catalina Patcher.

For the “Trim lib” part, here is an example of how I tried to solve the pyarrow dependence tree. To upload SuperChart to MAS (Mac App Store), I have to get rid of the dependence of Security.framework which introduced by pyarrow.

First, I tried to remove pyarrow from SuperChart. Later I found that it may cost a lot because it is not only involved in the parquet reading part but also in the superset serializing part. After some digging, I realized that the Security.framework seems to be some system-level OpenSSL in macOS. As we usually use pyarrow for serializing/deserializing or parquet reading, I don't think the crypto stuff is not really necessary for pyarrow.

Easy to know pyarrow is just a Python binding of Apache Arrow. So all I need to do is trim libArrow out of the Security.framework. Thanks to the well-implemented Arrow compiling system, after some flags are set. I got a very clean libarrow.500.dylib.

Frontend

For the frontend, there are 2 kinds of choices:

1. GUI Frameworks: PyQT, Tkinter, wxPython, Kivy, PySide (So much old and new frameworks)…

2. Web-Based Frameworks: Electron

Here are some useful comparisons or lists for “Python GUI Framework”

• Tkinter/PyQT/PySide/Kivy Comparison with Helloworld

• Some old but “official” list

• Yet another “Awesome XXX”, but with 128k stars

It seems there are much more options that can cause confusion. So I have to sort my specific needs instead of just Googling “Python GUI Framework” and choosing the most popular one. Here are my needs:

1. Superset is a really big project, I don’t want to rewrite the whole React stuff

2. I just wanna a macOS app, not a Linux, Windows, or any mobile platform

3. The smaller the better

Electron.js is really really popular, after simply:

ps aux | grep '(Renderer).app' | grep -v "Google Chrome.app"

I came across 3 Apps built with Electron.js that are currently running on my MacBook.

But as I know, nearly every OS is shipping with an XXWebView inside to make it easy for the App to open a web page. Why don't I just use that instead of packaging a Chromium which makes my package approximately 100MB larger?

So I got pywebview which let me open and control WKWebview to emulate an App UI. And also, thanks to my "the smaller the better" choice made it possible to put SuperChart to MAS. As Chromium uses some deprecated API of macOS which is not allowed for MAS.

Maybe the title could be "How I put a Python-based App onto Mac App Store".

SuperChart in MAS

Also published here.