Quickstart - Write a Package
Introduction
Welcome! This guide will walk you through how a Kurtosis package author would define their environment definition using Kurtosis. This guide takes ~15 minutes and will walk you through setting up a basic Postgres database and an API server to automate the loading of data. This guide is in a "code along" format, meaning we assume the user will be following the code examples and running Kurtosis CLI commands on your local machine. Everything you run in this guide is free, public, and does not contain any sensitive data.
This quickstart is meant for authors of environment definitions and is a continuation of the quickstart for package consumers. While you may choose to do the quickstarts in any order, it is recommended that you start with the quickstart for package consumers before this one. Doing both is highly recommended to understand how Kurtosis aims to solve the environment definition author-consumer divide to make building distributed systems as easy as building single server applications.
- Start a containerized Postgres database in Kurtosis
- Seed your database with test data using task sequencing
- Connect an API server to your database using dynamic service dependencies
- Parameterize your application setup in order to automate loading data into your API
- Make your Kurtosis package consumable by any user
TL;DR Version
This quickstart is in a "code along" format. You can also dive straight into running the end results and exploring the code too.
Open the Playground: Start
Click on the "New Workspace" button! You don't have to worry about the Context URL, Editor or Class. It's all pre-configured for you.
If you ever get stuck, every Kurtosis command accepts a -h flag to print helptext. If that doesn't help, you can get in touch with us in our Discord server or on Github!
Setup
Requirements
Before you proceed, please make sure you have:
- Installed and started the Docker engine
- Installed the Kurtosis CLI (or upgraded to latest if you already have it)
Hello, World
First, create and cd into a directory to hold the project you'll be working on:
mkdir kurtosis-postgres && cd kurtosis-postgres
Next, create a Starlark file called main.star inside your new directory with the following contents (more on Starlark in the "Review" section coming up soon):
def run(plan, args):
plan.print("Hello, world")
If you're using Visual Studio Code, you may find our Kurtosis VS Code Extension helpful when writing Starlark.
If you're using Vim, you can add the following to your .vimrc to get Starlark syntax highlighting:
" Add syntax highlighting for Starlark files
autocmd FileType *.star setlocal filetype=python
Finally, run the script (we'll explain enclaves in the "Review" section):
kurtosis run --enclave kurtosis-postgres main.star
Kurtosis will work for a bit, and then deliver you the following result:
INFO[2023-03-15T04:27:01-03:00] Creating a new enclave for Starlark to run inside...
INFO[2023-03-15T04:27:05-03:00] Enclave 'kurtosis-postgres' created successfully
> print msg="Hello, world"
Hello, world
Starlark code successfully run. No output was returned.
INFO[2023-03-15T04:27:05-03:00] ===================================================
INFO[2023-03-15T04:27:05-03:00] || Created enclave: kurtosis-postgres ||
INFO[2023-03-15T04:27:05-03:00] ===================================================
Name: kurtosis-postgres
UUID: a78f2ce1ca68
Status: RUNNING
Creation Time: Wed, 15 Mar 2023 04:27:01 -03
========================================= Files Artifacts =========================================
UUID Name
========================================== User Services ==========================================
UUID Name Ports Status
Congratulations - you've written your first Kurtosis code!
Review: Hello, World
You'll use these "Review" sections to explain what happened in the section.
In this section, you created a main.star file that simply told Kurtosis to print Hello, world. The .star extension corresponds to Starlark, a Python dialect also used by Google and Meta for configuring build systems.
When you ran main.star, you got Created enclave: quickstart. An enclave is a Kurtosis primitive that can be thought of as an ephemeral test environment, on top of Docker or Kubernetes, for a distributed application. The distributed applications that you define with Starlark will run inside enclaves. If you'd like, you can tear down your enclave and any of its artifacts by running: kurtosis clean -a (more on the kurtosis clean command here).
Enclaves are intended to be easy to create, easy to destroy, cheap to run, and isolated from each other. Use enclaves liberally!
Run Postgres
The heart of any application is the database. To introduce you to Kurtosis, we'll start by launching a Postgres server using Kurtosis.
Replace the contents of your main.star file with the following:
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
def run(plan, args):
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
),
)
Before you run the above command, remember that you still have the kurtosis-postgres enclave hanging around from the previous section. To clean up the previous enclave and execute our new main.star file above, run:
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres main.star
The --enclave flag is used to specify the enclave to use for that particular run. If one doesn't exist, Kurtosis will create an enclave with that name - which is what is happening here. Read more about kurtosis run here.
This entire "clean-and-run" process will be your dev loop for the rest of the kurtosis-postgres as you add more services and operations to our distributed application.
You'll see in the result that the kurtosis-postgres enclave now contains a Postgres instance:
Name: kurtosis-postgres
UUID: a30106a0bb87
Status: RUNNING
Creation Time: Tue, 14 Mar 2023 20:23:54 -03
========================================= Files Artifacts =========================================
UUID Name
========================================== User Services ==========================================
UUID Name Ports Status
b6fc024deefe postgres postgres: 5432/tcp -> postgresql://127.0.0.1:59299 RUNNING
Review: Run Postgres
So what actually happened? Three things actually:
- Interpretation: Kurtosis first ran your Starlark to build a plan for what you wanted done (in this case, starting a Postgres instance)
- Validation: Kurtosis then ran several validations against your plan, including validating that the Postgres image exists
- Execution: Kurtosis finally executed the validated plan inside the enclave to start a Postgres container
Note that Kurtosis did not execute anything until after Interpretation and Validation completed. You can think of Interpretation and Validation like Kurtosis' "compilation" step for your distributed application: you can catch many errors before any containers run, which shortens the dev loop and reduces the resource burden on your machine.
We call this approach multi-phase runs. While this approach has powerful benefits over traditional scripting, it also means you cannot reference Execution values like IP address in Starlark because they simply don't exist at Interpretation time. We'll explore how Kurtosis gracefully handles values generated during the Execution phase at the Interpretation phase later on in the quickstart.
This section introduced Kurtosis' ability to validate that definitions work as intended, before they are run - helping developers catch errors sooner & save resources when configuring multi-container test environments.
Add some data
A database without data is a fancy heater, so let's add some.
Your two options for seeding a Postgres database are:
- Making a sequence of PSQL commands via the
psqlbinary - Using
pg_restoreto load a package of data
Both are possible in Kurtosis, but for this tutorial we'll use pg_restore to seed your database with a TAR of DVD rental information, courtesy of postgresqltutorial.com.
Without Kurtosis
Normally going this route (using pg_restore) requires downloading the seed data to your local machine, starting Postgres, writing a pile of Bash to copy the seed data to the Postgres server, and then finally running the pg_restore command. If you forget to check if the database is available, you may get flakes when you try to use the seeding logic in a test.
Alternatively, you could use Docker Compose to volume-mount the data TAR into the Postgres server, but you'd still need to handle Postgres availability and sequencing the pg_restore afterwards.
With Kurtosis
By contrast, Kurtosis Starlark scripts can use data as a first-class primitive and sequence tasks such as pg_restore into the plan.
Let's see it in action, and we'll explain what's happening afterwards.
First, create a file called kurtosis.yml next to your main.star file (in your working directory, kurtosis-postgres) with the following contents:
name: "github.com/john-snow/kurtosis-postgres"
Then update your main.star with the following:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
SEED_DATA_DIRPATH = "/seed-data"
def run(plan, args):
# Make data available for use in Kurtosis
data_package_module_result = data_package_module.run(plan, {})
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
# Load the data into Postgres
postgres_flags = ["-U", POSTGRES_USER,"-d", POSTGRES_DB]
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]),
)
Now, run the following to see what happens:
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres .
(Notice you are using . instead of main.star)
The output should also look more interesting as your plan has grown bigger:
INFO[2023-03-15T04:34:06-03:00] Cleaning enclaves...
INFO[2023-03-15T04:34:06-03:00] Successfully removed the following enclaves:
60601dd9906e40d6af5f16b233a56ae7 kurtosis-postgres
INFO[2023-03-15T04:34:06-03:00] Successfully cleaned enclaves
INFO[2023-03-15T04:34:06-03:00] Cleaning old Kurtosis engine containers...
INFO[2023-03-15T04:34:06-03:00] Successfully cleaned old Kurtosis engine containers
INFO[2023-03-15T04:34:06-03:00] Creating a new enclave for Starlark to run inside...
INFO[2023-03-15T04:34:10-03:00] Enclave 'kurtosis-postgres' created successfully
INFO[2023-03-15T04:34:10-03:00] Executing Starlark package at '/tmp/kurtosis-postgres' as the passed argument '.' looks like a directory
INFO[2023-03-15T04:34:10-03:00] Compressing package 'github.com/john-snow/kurtosis-postgres' at '.' for upload
INFO[2023-03-15T04:34:10-03:00] Uploading and executing package 'github.com/john-snow/kurtosis-postgres'
> upload_files src="github.com/kurtosis-tech/awesome-kurtosis/data-package/dvd-rental-data.tar"
Files with artifact name 'howling-thunder' uploaded with artifact UUID '32810fc8c131414882c52b044318b2fd'
> add_service name="postgres" config=ServiceConfig(image="postgres:15.2-alpine", ports={"postgres": PortSpec(number=5432, application_protocol="postgresql")}, files={"/seed-data": "howling-thunder"}, env_vars={"POSTGRES_DB": "app_db", "POSTGRES_PASSWORD": "password", "POSTGRES_USER": "app_user"})
Service 'postgres' added with service UUID 'f1d9cab2ca344d1fbb0fc00b2423f45f'
> exec recipe=ExecRecipe(command=["pg_restore", "-U", "app_user", "-d", "app_db", "--no-owner", "--role=app_user", "/seed-data/dvd-rental-data.tar"])
Command returned with exit code '0' with no output
Starlark code successfully run. No output was returned.
INFO[2023-03-15T04:34:21-03:00] ===================================================
INFO[2023-03-15T04:34:21-03:00] || Created enclave: kurtosis-postgres ||
INFO[2023-03-15T04:34:21-03:00] ===================================================
Name: kurtosis-postgres
UUID: 995fe0ca69fe
Status: RUNNING
Creation Time: Wed, 15 Mar 2023 04:34:06 -03
========================================= Files Artifacts =========================================
UUID Name
32810fc8c131 howling-thunder
========================================== User Services ==========================================
UUID Name Ports Status
f1d9cab2ca34 postgres postgres: 5432/tcp -> postgresql://127.0.0.1:62914 RUNNING
Does your Postgres have data now? Let's find out by opening a shell on the Postgres container and logging into the database:
kurtosis service shell kurtosis-postgres postgres
From there, listing the tables in the Postgres can be done with:
psql -U app_user -d app_db -c '\dt'
...which will reveal that many new tables now exist:
List of relations
Schema | Name | Type | Owner
--------+---------------+-------+----------
public | actor | table | app_user
public | address | table | app_user
public | category | table | app_user
public | city | table | app_user
public | country | table | app_user
public | customer | table | app_user
public | film | table | app_user
public | film_actor | table | app_user
public | film_category | table | app_user
public | inventory | table | app_user
public | language | table | app_user
public | payment | table | app_user
public | rental | table | app_user
public | staff | table | app_user
public | store | table | app_user
(15 rows)
Feel free to explore the Postgres container. When you're done, run either exit or press Ctrl-D.
Review: Add some data
So what just happened?
We created a Kurtosis package
By creating a kurtosis.yml file in your working directory, you turned your working directory into a Kurtosis package (specifically, a runnable package). After you did this, your newly created Kurtosis package could now declare dependencies on external packages using Kurtosis’ built-in packaging/dependency system.
To see this in action, the first line in your local main.star file was used to import, and therefore declare a dependency on, an external package called data-package using a locator:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
... which we then ran locally:
data_package_module_result = data_package_module.run(plan, {})
This external Kurtosis package, named "data-package" contains the seed data for your Postgres instance that we referenced earlier as a .tar file.
Special note here that we used a locator to import an external package from your awesome-kurtosis repository and not a regular URL. Learn more about how they differ here.
You imported seed data into your Kurtosis package
The main.star file in that external "data-package" contained Starlark instructions to store the .tar data as a files artifact using the files_upload Starlark instruction:
TAR_FILENAME = "dvd-rental-data.tar"
def run(plan, args):
dvd_rental_data = plan.upload_files("github.com/kurtosis-tech/awesome-kurtosis/data-package/" + TAR_FILENAME)
result = struct(
files_artifact = dvd_rental_data, # Needed to mount the data on a service
tar_filename = TAR_FILENAME, # Useful to reference the data TAR contained in the files artifact
)
return result
A files artifact is Kurtosis' first-class data primitive and is a TGZ of arbitrary files living inside an enclave. So long as a files artifact exists, Kurtosis knows how to mount its contents on a service.
You mounted and seeded the data into your Postgres instance
Next, you mounted the seed data, stored in your enclave now as a files artifact, into your Postgres instance using the ServiceConfig.files option:
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
# ...omitted...
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
Then to seed the data, you used the exec Starlark instruction:
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]
),
Here, you saw one of Kurtosis' most loved features: the ability to modularize and share your distributed application logic using only a Github repository. We won't dive into all the usecases now, but the examples here can serve as a good source of inspiration.
Add an API
Databases don't come alone, however. In this section we'll add a PostgREST API in front of the database and see how Kurtosis handles inter-service dependencies.
Replace the contents of your main.star with this:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
SEED_DATA_DIRPATH = "/seed-data"
POSTGREST_PORT_ID = "http"
def run(plan, args):
# Make data available for use in Kurtosis
data_package_module_result = data_package_module.run(plan, {})
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
# Load the data into Postgres
postgres_flags = ["-U", POSTGRES_USER,"-d", POSTGRES_DB]
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]),
)
# Add PostgREST
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
"postgres",
POSTGRES_PASSWORD,
postgres.ip_address,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
api = plan.add_service(
name = "api", # Naming our PostgREST service "api"
config = ServiceConfig(
image = "postgrest/postgrest:v10.2.0.20230209",
env_vars = {
"PGRST_DB_URI": postgres_url,
"PGRST_DB_ANON_ROLE": POSTGRES_USER,
},
ports = {POSTGREST_PORT_ID: PortSpec(3000, application_protocol = "http")},
)
)
Now, run the same dev loop command as before (and don't worry about the result, we'll explain that later):
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres .
We just got a failure, just like we might when building a real system!
> add_service name="api" config=ServiceConfig(image="postgrest/postgrest:v10.2.0.20230209", ports={"http": PortSpec(number=3000, application_protocol="http")}, env_vars={"PGRST_DB_ANON_ROLE": "app_user", "PGRST_DB_URI": "postgresql://postgres:password@{{kurtosis:4d65eca66b5749df8988419ae31dda21:ip_address.runtime_value}}:5432/app_db"})
There was an error executing Starlark code
An error occurred executing instruction (number 4) at DEFAULT_PACKAGE_ID_FOR_SCRIPT[54:27]:
add_service(name="api", config=ServiceConfig(image="postgrest/postgrest:v10.2.0.20230209", ports={"http": PortSpec(number=3000, application_protocol="http")}, env_vars={"PGRST_DB_ANON_ROLE": "app_user", "PGRST_DB_URI": "postgresql://postgres:password@{{kurtosis:4d65eca66b5749df8988419ae31dda21:ip_address.runtime_value}}:5432/app_db"}))
Caused by: Unexpected error occurred starting service 'api'
Caused by: An error occurred waiting for all TCP and UDP ports being open for service 'api' with private IP '10.1.0.4'; as the most common error is a wrong service configuration, here you can find the service logs:
== SERVICE 'api' LOGS ===================================
09/May/2023:19:18:41 +0000: Attempting to connect to the database...
09/May/2023:19:18:41 +0000: {"code":"PGRST000","details":"connection to server at \"10.1.0.3\", port 5432 failed: FATAL: password authentication failed for user \"postgres\"\n","hint":null,"message":"Database connection error. Retrying the connection."}
09/May/2023:19:18:41 +0000: connection to server at "10.1.0.3", port 5432 failed: FATAL: password authentication failed for user "postgres"
postgrest: thread killed
== FINISHED SERVICE 'api' LOGS ===================================
Caused by: An error occurred while waiting for all TCP and UDP ports to be open
Caused by: Unsuccessful ports check for IP '10.1.0.4' and port spec '{number:3000 transportProtocol:0 applicationProtocol:0xc006662e10 wait:0xc00662d510}', even after '2' retries with '500' milliseconds in between retries. Timeout '15s' has been reached
Caused by: An error occurred while calling network address '10.1.0.4:3000' with port protocol 'TCP' and using time out '14.499139733s'
Caused by: dial tcp 10.1.0.4:3000: i/o timeout
Error encountered running Starlark code.
Here, Kurtosis is telling us that the add_service instruction on line 54 of your main.star (the one for ensuring PostgREST is up) is timing out when was checking for ports opening.
Fun fact: this failure was encountered at the last step in Kurtosis' multi-phase run approach, which is also called the Execution step that we mentioned earlier when we got Postgres up and running.
Investigating the issue
If you check the service's logs, printed in the error message right after this header == SERVICE 'api' LOGS ===================================, you will see that there is an authentication error
The enclave state is usually a good place to find mor clues. If you look at the bottom of your output you'll see the following state of the enclave:
Name: kurtosis-postgres
UUID: 5b360f940bcc
Status: RUNNING
Creation Time: Tue, 14 Mar 2023 22:15:19 -03
========================================= Files Artifacts =========================================
UUID Name
323c9a71ebbf crimson-haze
========================================== User Services ==========================================
UUID Name Ports Status
45b355fc810b postgres postgres: 5432/tcp -> postgresql://127.0.0.1:59821 RUNNING
From the above, we can see that the PostgREST service (named: api) is not in the 'User Services' list, so we can infer that it crashed when it was starting.
You can also grab the PostgREST logs...
kurtosis service logs kurtosis-postgres api
...we can see that the PostgREST is dying:
15/Mar/2023:01:15:30 +0000: Attempting to connect to the database...
15/Mar/2023:01:15:30 +0000: {"code":"PGRST000","details":"FATAL: password authentication failed for user \"postgres\"\n","hint":null,"message":"Database connection error. Retrying the connection."}
15/Mar/2023:01:15:30 +0000: FATAL: password authentication failed for user "postgres"
postgrest: thread killed
Looking back to your Starlark code, you can see the problem: it's creating the Postgres database with a user called app_user, but it's telling PostgREST to try and connect through a user called postgres:
POSTGRES_USER = "app_user"
# ...
def run(plan, args):
# ...
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
# ...
env_vars = {
# ...
"POSTGRES_USER": POSTGRES_USER,
# ...
},
# ...
),
)
# ...
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
"postgres", # <---------- THE PROBLEM IS HERE
POSTGRES_PASSWORD,
postgres.ip_address,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
In the line declaring the postgres_url variable in your main.star file, replace the "postgres", string with POSTGRES_USER, to use the correct username we specified at the beginning of our file. Then re-run your dev loop:
kurtosis clean -a && kurtosis run --enclave kurtosis-postgres .
Now at the bottom of the output we can see that the PostgREST service is RUNNING correctly:
Name: kurtosis-postgres
UUID: 11c0ac047299
Status: RUNNING
Creation Time: Tue, 14 Mar 2023 22:30:02 -03
========================================= Files Artifacts =========================================
UUID Name
323c9a71ebbf crimson-haze
========================================== User Services ==========================================
UUID Name Ports Status
ce90b471a982 postgres postgres: 5432/tcp -> postgresql://127.0.0.1:59883 RUNNING
98094b33cd9a api http: 3000/tcp -> http://127.0.0.1:59887 RUNNING
Review: Add an API
In this section, you spun up a new PostgREST service (that we named api for readability) with a dependency on the Postgres service. Normally, PostgREST needs to know the IP address or hostname of the Postgres service, and we said earlier that Starlark (the Interpretation phase) can never know Execution values.
So how did the services get connected?
Answer: Execution-time values are represented at Interpretation time as future references that Kurtosis will replace at Execution time with the actual value. In this case, the postgres_url variable here...
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
POSTGRES_USER,
POSTGRES_PASSWORD,
postgres.ip_address,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
...used the postgres.ip_address and postgres.ports[POSTGRES_PORT_ID].number future references returned by adding the Postgres service, so that when postgres_url was used as an environment variable during PostgREST startup...
api = plan.add_service(
name = "api", # Naming our PostgREST service "api"
config = ServiceConfig(
# ...
env_vars = {
"PGRST_DB_URI": postgres_url, # <-------- HERE
"PGRST_DB_ANON_ROLE": POSTGRES_USER,
},
# ...
)
)
...Kurtosis simply swapped in the correct Postgres container Execution-time values. While future references take some getting used to, we've found the feedback loop speedup to be very worth it.
What you've just seen is Kurtosis' powerful ability to gracefully handle data generated at runtime to set up service dependencies in multi-container test environments. You also saw how seamless it was to run on-box CLI commands on a container.
Modifying data
Now that you have an API, you should be able to interact with the data.
Inspect your enclave:
kurtosis enclave inspect kurtosis-postgres
Notice how Kurtosis automatically exposed the PostgREST container's http port to your machine:
28a923400e50 api http: 3000/tcp -> http://127.0.0.1:59992 RUNNING
In this output the http port is exposed as URL http://127.0.0.1:59992, but your port number will be different.
You can paste the URL from your output into your browser (or Cmd+click if you're using iTerm) to verify that you are indeed talking to the PostgREST inside your kurtosis-postgres enclave:
{"swagger":"2.0","info":{"description":"","title":"standard public schema","version":"10.2.0.20230209 (pre-release) (a1e2fe3)"},"host":"0.0.0.0:3000","basePath":"/","schemes":["http"],"consumes":["application/json","application/vnd.pgrst.object+json","text/csv"],"produces":["application/json","application/vnd.pgrst.object+json","text/csv"],"paths":{"/":{"get":{"tags":["Introspection"],"summary":"OpenAPI description (this document)","produces":["application/openapi+json","application/json"],"responses":{"200":{"description":"OK"}}}},"/actor":{"get":{"tags":["actor"],"parameters":[{"$ref":"#/parameters/rowFilter.actor.actor_id"},{"$ref":"#/parameters/rowFilter.actor.first_name"},{"$ref":"#/parameters/rowFilter.actor.last_name"},{"$ref":"#/parameters/rowFilter.actor.last_update"},{"$ref":"#/parameters/select"},{"$ref":"#/parameters/order"},{"$ref":"#/parameters/range"},{"$ref":"#/parameters/rangeUnit"},{"$ref":"#/parameters/offset"},{"$ref":"#/parameters/limit"},{"$ref":"#/parameters/preferCount"}], ...
Now make a request to insert a row into the database (replacing $YOUR_PORT with the http port from your enclave inspect output for the PostgREST service that we named api)...
curl -XPOST -H "content-type: application/json" http://127.0.0.1:$YOUR_PORT/actor --data '{"first_name": "Kevin", "last_name": "Bacon"}'
...and then query for it (again replacing $YOUR_PORT with your port)...
curl -XGET "http://127.0.0.1:$YOUR_PORT/actor?first_name=eq.Kevin&last_name=eq.Bacon"
...to get it back:
[{"actor_id":201,"first_name":"Kevin","last_name":"Bacon","last_update":"2023-03-15T02:08:14.315732"}]
Of course, it'd be much nicer to formalize this in Kurtosis. Replace your main.star with the following:
data_package_module = import_module("github.com/kurtosis-tech/awesome-kurtosis/data-package/main.star")
POSTGRES_PORT_ID = "postgres"
POSTGRES_DB = "app_db"
POSTGRES_USER = "app_user"
POSTGRES_PASSWORD = "password"
SEED_DATA_DIRPATH = "/seed-data"
POSTGREST_PORT_ID = "http"
def run(plan, args):
# Make data available for use in Kurtosis
data_package_module_result = data_package_module.run(plan, {})
# Add a Postgres server
postgres = plan.add_service(
name = "postgres",
config = ServiceConfig(
image = "postgres:15.2-alpine",
ports = {
POSTGRES_PORT_ID: PortSpec(5432, application_protocol = "postgresql"),
},
env_vars = {
"POSTGRES_DB": POSTGRES_DB,
"POSTGRES_USER": POSTGRES_USER,
"POSTGRES_PASSWORD": POSTGRES_PASSWORD,
},
files = {
SEED_DATA_DIRPATH: data_package_module_result.files_artifact,
}
),
)
# Load the data into Postgres
postgres_flags = ["-U", POSTGRES_USER,"-d", POSTGRES_DB]
plan.exec(
service_name = "postgres",
recipe = ExecRecipe(command = ["pg_restore"] + postgres_flags + [
"--no-owner",
"--role=" + POSTGRES_USER,
SEED_DATA_DIRPATH + "/" + data_package_module_result.tar_filename,
]),
)
# Add PostgREST
postgres_url = "postgresql://{}:{}@{}:{}/{}".format(
POSTGRES_USER,
POSTGRES_PASSWORD,
postgres.hostname,
postgres.ports[POSTGRES_PORT_ID].number,
POSTGRES_DB,
)
api = plan.add_service(
name = "api",
config = ServiceConfig(
image = "postgrest/postgrest:v10.2.0.20230209",
env_vars = {
"PGRST_DB_URI": postgres_url,
"PGRST_DB_ANON_ROLE": POSTGRES_USER,
},
ports = {POSTGREST_PORT_ID: PortSpec(3000, application_protocol = "http")},
)
)
# Insert data
if "actors" in args:
insert_data(plan, args["actors"])
def insert_data(plan, data):
plan.request(
service_name = "api",
recipe = PostHttpRequestRecipe(
port_id = POSTGREST_PORT_ID,
endpoint = "/actor",
content_type = "application/json",
body = json.encode(data),
)
)
Now clean and run, only this time with extra args to kurtosis run:
kurtosis clean -a && kurtosis run --enclave quickstart . '{"actors": [{"first_name":"Kevin", "last_name": "Bacon"}, {"first_name":"Steve", "last_name":"Buscemi"}]}'
Using the new http URL on the api service in the output, query for the rows you just added (replacing $YOUR_PORT with your correct PostgREST http port number)...
curl -XGET "http://127.0.0.1:$YOUR_PORT/actor?or=(last_name.eq.Buscemi,last_name.eq.Bacon)"
...to yield:
[{"actor_id":201,"first_name":"Kevin","last_name":"Bacon","last_update":"2023-03-15T02:29:53.454697"},
{"actor_id":202,"first_name":"Steve","last_name":"Buscemi","last_update":"2023-03-15T02:29:53.454697"}]
Review
How did this work?
Mechanically, we first created a JSON string of data using Starlark's json.encode builtin. Then we used the request Starlark instruction to shove the string at PostgREST, which writes it to the database:
plan.request(
service_name = "api",
recipe = PostHttpRequestRecipe(
port_id = POSTGREST_PORT_ID,
endpoint = "/actor",
content_type = "application/json",
body = json.encode(data),
)
)
At a higher level, Kurtosis automatically deserialized the {"actors": [{"first_name":"Kevin", "last_name": "Bacon"}, {"first_name":"Steve", "last_name":"Buscemi"}]} string passed as a parameter to kurtosis run, and put the deserialized object in the args parameter to the run function in main.star:
def run(plan, args):
This section showed how to interact with your test environment, and also how to parametrize it for others to easily modify and re-use.
Publishing your Kurtosis Package for others to use
At this point, you should have 2 files:
main.starkurtosis.yml
Publishing these 2 files to a Github repository will enable anyone with Kurtosis and an internet connection to instantiate the environment you described in your main.star file. These files, alongside a directory that you will create on Github, form the basis for a Kurtosis package
Once you've uploaded it to a Github repository, anyone can spin up the same system - in any way they want (using your parameters) and anywhere they want (e.g. Docker, Kubernetes, local, remote, etc).
As an example, a consumer of the Kurtosis package need only run the following command to reproduce the system and add another row to the database of actors:
kurtosis run github.com/YOUR_GITHUB_USERNAME/REPOSITORY_NAME '{"actors": [{"first_name":"Harrison", "last_name": "Ford"}
Where YOUR_GITHUB_USERNAME is your Github username, and REPOSITORY_NAME is the name of your Github repository that houses your kurtosis.yml and main.star files.
Conclusion
And that's it - you've written your very first distributed application in Kurtosis! You experienced a great many workflows that Kurtosis optimizes to help bridge the author-consumer divide for environment definitions. Namely, as an author, you:
- Started a Postgres database in an ephemeral, isolated test environment.
- Seeded your database by importing an external Starlark package from the internet.
- Set up an API server for your database and gracefully handled dynamically generated dependency data.
- Inserted & queried data via the API.
- Parameterized data insertion for future use by other users.
- Made your Kurtosis package consumable by any user by publishing a Kurtosis package to Github
This was still just an introduction to Kurtosis. To dig deeper, visit other sections of our docs where you can read about what Kurtosis is, understand the architecture, and hear our inspiration for starting Kurtosis.
Finally, we'd love to hear from you. Please don't hesitate to share with us what went well, and what didn't, using kurtosis feedback to file an issue in our Github or to chat with our cofounder, Kevin.
Lastly, feel free to star us on Github, join the community in our Discord, and follow us on Twitter!