Long story short, there is no official git integration available for your streamlit apps on Snowflake. Here is an example how this could be done and what are the minimum requirements for uploading your apps to Snowflake stage.
Adding files to the stage can be done using SnowSQL documentation.
Prerequisites
To get started, we need to have a few things set up:
You need to have a user for logging in to Snowflake and having access too a database, schema, stage and a warehouse. Here is a list of things you need to run for you user that you will be using to run your Github Action:
Create user and ROLE
CREATE USER STREAMLIT_DEVELOPER;
CREATE ROLE STREAMLIT_DEVOPS;
Grant rights to work with streamlit
GRANT USAGE ON SCHEMA STREAMLIT_DB.STREAMLIT_SCHEMA TO ROLE STREAMLIT_DEVOPS;
GRANT USAGE ON DATABASE STREAMLIT_DEV TO ROLE STREAMLIT_CREATOR;
GRANT CREATE STREAMLIT ON SCHEMA STREAMLIT_DEV.STREAMLIT TO ROLE STREAMLIT_DEVOPS;
GRANT CREATE STAGE ON SCHEMA STREAMLIT_DEV.STREAMLIT TO ROLE STREAMLIT_DEVOPS;
Secrets needed
Now that you have done that add the variables to Github secrets. In this demo you will need these variables. Some are more sensitive then other so let’s keep them as a secret.
SNOWSQL_ACCOUNT: ${{ secrets.ACCOUNT_REGION_PLATFORM }}
SNOWSQL_USER: ${{ secrets.SNOWFLAKE_USERNAME }}
SNOWSQL_PWD: ${{ secrets.SNOWFLAKE_PASSWORD }}
SNOWSQL_SCHEMA: "STREAMLIT_DEV"
SNOWSQL_WAREHOUSE: "COMPUTE_WH"
SNOWSQL_DATABASE: "STREAMLIT_DEV"
SNOWSQL_STAGE: "DEMO1"
SNOWSQL_ROLE: ${{ secrets.SNOWFLAKE_ROLE }}
Implementation
Considering that you might have several different apps in the folder and want them all to be deployed into Snowflake. I have made a loop that will loop the streamlit folder. The folder name will be also used as a stage name. Also the landing app file is always the same, I’m using app.py.
Here is an example loop that can be used to add the files to the stage:
for file_path in $(find streamlit -type f); do
stage_name=$(echo $file_path | cut -d'/' -f2)
~/bin/snowsql -q "put file://$file_path @$stage_name overwrite=true auto_compress=false;"
echo "Added file: $file_path to stage $stage_name"
done
After you have done that, you need to create the app itself also, here is how you can do that:
for stage_name in $(find streamlit -mindepth 1 -maxdepth 1 -type d -print | sed 's,streamlit/,,'); do
~/bin/snowsql -q "CREATE OR REPLACE STREAMLIT $SNOWSQL_DATABASE.$SNOWSQL_SCHEMA.$stage_name ROOT_LOCATION = '@$SNOWSQL_DATABASE.$SNOWSQL_SCHEMA.$stage_name' MAIN_FILE = '/app.py' QUERY_WAREHOUSE = 'COMPUTE_WH';"
echo "Create streamlit: $stage_name"
done
The build Yaml for Deployment
The build Yaml can be found from here
The job consists of several steps that perform various tasks. The first step checks out the code from the repository. The second step installs SnowSQL, a command-line tool for interacting with Snowflake. The third step creates a profile for SnowSQL that includes the connection details for the Snowflake account, such as the account name, username, password, schema, warehouse, database, and role. The fourth step logs in to the Snowflake account using the profile created in the previous step.
The fifth step sets the destination database and schema for the SnowSQL session. The sixth step creates a Snowflake stage for each subdirectory in the “streamlit” directory of the repository. The seventh step uploads all files in the “streamlit” directory to their corresponding stages in Snowflake. The eighth step creates a Snowflake Streamlit object for each subdirectory in the “streamlit” directory, specifying the root location of the stage, the main Python file, and the query warehouse.
Conclusion
This is a way to have a version 0.1, which would provide you a way to keep track of the changes and correct the mistakes and develop your apps so that you can reverse the code if needed.
Managing change in the data base, it could be wise to use Terraform for granting the rights to the apps and also deleting unused streamlit apps.
Happy coding"
The code can be found from github Github link