Executing Cognify#

Cognify provides a simple command-line interface (CLI) to optimize, evaluate, and inspect generative AI workflows.

Basic Syntax#

cognify [mode_switch] <path_to_workflow> [optional common_options] [optional mode_options]

Following options that are common across all modes:

Short	Long	Parameter	Default	Description
-c	--config	<config filepath>	See Description	Specify path to the configuration file for the workflow. If not provided, will look for `config.py` in the same directory as the workflow script.
-l	--log_level	DEBUG, INFO, WARNING, ERROR, CRITICAL	WARNING	Sets the logging level

Mode	Description
optimize	Searches for cost-effective configurations of the given workflow. All intermediate results are saved for resuming.
evaluate	Evaluates the a specific configuration on the test data.
inspect	Lists all the Pareto-frontiers found during optimization so far.

cognify optimize <path_to_workflow> [optional common_options] [optional optimize_options]

Short	Long	Parameter	Default	Description
-r	--resume		False	Resume optimization using saved intermediate results.
-f	--force		False	Force overwriting the existing result folder.

cognify evaluate <path_to_workflow> [optional common_options] [optional evaluate_options]

Short	Long	Parameter	Default	Description
-s	--select	Optimization_1, Optimization_2, …, or Original	Original	Select a specific configuration by ID for evaluation. Use `Original` to evaluate the original workflow.
-j	--n_parallel	1, 2, …	10	Number of parallel data points to evaluate. Refer to the API rate limit of your model provider when setting this value.
-o	--output_path	<filename>	See Description	Path to save evaluation results. Defaults to `eval_result.json` in the workflow script’s directory.

cognify inspect <path_to_workflow> [optional common_options] [optional inspect_options]

Short	Long	Parameter	Default	Description
-f	--dump_optimization_results		False	Dump descriptive details of all optimizations found `optimized_workflow_details` directory in the optimization result folder.

Check help info

cognify -h

check help info for optimize mode

cognify optimize -h

Optimize a workflow

cognify optimize workflow.py

If config file is named differently and want to overwite the result folder:

cognify optimize workflow.py -c <path_to_config> -f

Evaluate a configuration

cognify evaluate workflow.py --select Optimization_1

Evaluate the original program with a batch size of 50:

cognify evaluate workflow.py -j 50

Inspect current optimization results

cognify inspect workflow.py --dump_optimization_results