Command Line Tools

The beautifuljason.tools package provides a set of ready-to-use command-line tools to support batch processing and configuration tasks for the JASON desktop application and JJH5 documents.

These tools are fully functional, ready to use, and designed to be run directly from the terminal. They allow users to automate workflows, manage configurations, and process data efficiently.

jason_config

Manage JASON installation paths used by BeautifulJASON.

Usage:

jason_config [--display] [--add_path PATH] [--del_path INDEX]
             [--set_preferred INDEX] [--reset]

Description:

Allows inspection and modification of the list of known JASON installation paths used by the library. Includes features for setting preferred paths, resetting configuration, and verifying installation validity.

Typical use cases:

  • Configuring BeautifulJASON to point to a specific JASON installation

  • Resetting or managing existing JASON paths

  • Debugging JASON configuration issues

Usage examples:

  1. Display the Current Configuration:

    To view the current path settings for the JASON application:

    jason_config --display

  2. Add a New JASON Path:

    If the JASON application resides in a different location than the detected default paths:

    jason_config --add_path /path/to/your/jason/application

    Be sure to replace /path/to/your/jason/application with the actual path to your JASON executable.

  3. For Additional Commands and Options:

    If you need more details about available commands or want to explore other options:

    jason_config --help

Always ensure that the specified path points directly to the JASON executable for BeautifulJASON to function correctly.

jason_batch_convert

Batch convert input files to various formats.

Usage:

jason_batch_convert [-h] in_dir out_dir --formats FORMATS [FORMATS ...]
                    (--extensions EXTENSIONS [EXTENSIONS ...] | --patterns PATTERNS [PATTERNS ...])
                    [--rules RULES] [--execute]

Description:

Recursively scans the input directory for files matching specified extensions or path patterns and converts them to the selected formats. The tool can operate in dry mode (default) to list the matching files or in execution mode using the --execute option to perform the actual conversion.

By leveraging the --rules option, users can integrate automatic processing, analysis, and layout generation for reporting purposes. This makes the tool an essential first step in preparing data for advanced analysis workflows.

Formats supported: jjh5, jjj, jdx, jdf, pdf, png, jpg, svg

Useful for: automation of data conversion pipelines using BeautifulJASON and JASON.

Usage examples:

  1. Dry Run to List Matching Files:

    To scan the input directory for .jdf files and list the matching files without performing any conversion:

    jason_batch_convert ./input ./output --formats pdf --extensions jdf

  2. Convert Files to Multiple Formats:

    To convert all .jdf files in the input directory to both pdf and jjh5 formats:

    jason_batch_convert ./input ./output --formats pdf jjh5 --extensions jdf --execute

  3. Use Path Patterns for File Selection:

    To convert files matching a specific pattern (e.g., files starting with sample_) to pdf format:

    jason_batch_convert ./input ./output --formats pdf --patterns sample_* --execute

  4. Convert Files in Specific Directory Structures:

    To process datasets organized in specific directory structures, such as fid files located in subdirectories named 10, and convert them to jjh5 format:

    jason_batch_convert ./input ./output --formats jjh5 --patterns */10/fid --execute

    This example matches fid files located in subdirectories named 10 within the input directory. The */10/fid pattern ensures that the tool looks for fid files specifically in directories named 10, regardless of their parent directory. The */ part acts as a wildcard, allowing the tool to search through all subdirectories of the input directory to find matching paths.

  5. Apply Custom Rules During Conversion:

    To apply a specific set of processing rules during conversion:

    jason_batch_convert ./input ./output --formats pdf --extensions jdf --rules custom_rules.jjr --execute

Source code:

The source code for this tool is provided for users to review, explore, and gain insights into its implementation. However, the tool itself is fully functional and ready to use as-is.

  1##
  2##-----------------------------------------------------------------------------
  3##
  4## Copyright (c) 2024 JEOL Ltd.
  5## 1-2 Musashino 3-Chome
  6## Akishima Tokyo 196-8558 Japan
  7##
  8## This software is provided under the MIT License. For full license information,
  9## see the LICENSE file in the project root or visit https://opensource.org/licenses/MIT
 10##
 11##++---------------------------------------------------------------------------
 12##
 13## ModuleName : BeautifulJASON
 14## ModuleType : Python API for JASON desktop application and JJH5 documents
 15## Purpose : Automate processing, analysis, and report generation with JASON
 16## Author : Nikolay Larin
 17## Language : Python
 18##
 19####---------------------------------------------------------------------------
 20##
 21
 22import os
 23import datetime
 24from multiprocessing import Pool, Manager
 25from multiprocessing.managers import ValueProxy
 26from pathlib import Path
 27import argparse
 28import glob
 29from colorama import Fore, init
 30
 31import beautifuljason as bjason
 32
 33def unique_output_filename(file, extension):
 34    """Generate a unique output filename based on the input file path and desired extension."""
 35    unique_base_name = file.replace(os.sep, '_')  # Replace path separators with underscores
 36    return unique_base_name + '.' + extension
 37
 38def process_file(execute: bool, file: str, in_dir: str, out_dir: str, jason: bjason.JASON, formats: list[str], rules: str, counter: ValueProxy, total_files: int):
 39    """Process a single file, converting it to specified formats."""
 40    out_fnames = [unique_output_filename(file, format) for format in formats]
 41    if execute:
 42        global processed_files
 43        with jason.create_document(os.path.join(in_dir, file), rules=rules) as doc:
 44            doc.close()
 45            jason.save(doc, [os.path.join(out_dir, fname) for fname in out_fnames])
 46    counter.value += 1
 47    print(f"{counter.value}/{total_files}: {file} => {', '.join(out_fnames)}")
 48
 49def main():
 50    init(autoreset=True)  # Initialize colorama
 51
 52    parser = argparse.ArgumentParser(
 53        description='Convert files from a specified directory based on extensions or patterns.',
 54        usage='jason_batch_convert [-h] in_dir out_dir --formats FORMATS [FORMATS...] (--extensions EXTENSIONS [EXTENSIONS ...] | --patterns PATTERNS [PATTERNS ...]) [--rules RULES] [--execute]'
 55        )
 56    parser.add_argument('in_dir', help='Root directory containing files to be converted.')
 57    parser.add_argument('out_dir', help='Directory where the converted files will be saved.')
 58    parser.add_argument('--formats', nargs='+', required=True, 
 59                        choices=['jjh5', 'jjj', 'jdx', 'jdf', 'pdf', 'png', 'jpg', 'svg'],
 60                        help='List of desired output file formats. Multiple formats can be specified.')
 61    parser.add_argument('--rules', type=str, default="off",
 62                        help='Path to the rules file or rule library name.')
 63    parser.add_argument('--execute', action='store_true', 
 64                        help='Execute the file conversions. If not specified, will perform a dry run, listing the files to be converted. It is recommended to run a dry run first.')
 65
 66    group = parser.add_mutually_exclusive_group(required=True)
 67    group.add_argument('--extensions', nargs='+', help='List of file extensions to convert, e.g., jdf, als, jdx, mol, etc.')
 68    group.add_argument('--patterns', nargs='+', help='List of file path patterns to convert, e.g., */*.fid/fid, */10/fid, etc. Patterns cannot end with "/*".')
 69
 70    args = parser.parse_args()
 71
 72    # Convert in_dir and out_dir to absolute paths
 73    args.in_dir = os.path.abspath(args.in_dir)
 74    args.out_dir = os.path.abspath(args.out_dir)
 75
 76    if not args.execute:
 77        print(f"{Fore.YELLOW}Dry run mode! No files will be converted.")
 78    print("Input directory:", args.in_dir)
 79    print("Output directory:", args.out_dir)
 80
 81    # Initialize Jason
 82    jason = bjason.JASON()
 83
 84    # Create output directory
 85    Path(args.out_dir).mkdir(parents=True, exist_ok=True)
 86
 87    # Find and process files matching patterns or extensions
 88    print("Searching for files...")
 89    files_to_process = []
 90    if args.patterns:
 91        for pattern in args.patterns:
 92            if pattern.endswith("/*"):
 93                raise ValueError("Invalid pattern. Patterns cannot end with '/*'.")
 94            pattern_files = glob.glob(os.path.join(args.in_dir, pattern), recursive=True)
 95            files_to_process.extend([os.path.relpath(f, start=args.in_dir) for f in pattern_files if os.path.isfile(f)])
 96    else:
 97        for root, dirs, files in os.walk(args.in_dir):
 98            for file in files:
 99                if file.split('.')[-1] in args.extensions:
100                    full_path = os.path.join(root, file)
101                    if os.path.isfile(full_path):
102                        files_to_process.append(os.path.relpath(full_path, start=args.in_dir))
103
104    total_files = len(files_to_process)
105    print(f"{total_files} files found in the input directory.")
106
107    manager = Manager()
108    counter = manager.Value('i', 0)  # integer counter initialized to 0
109
110    if args.execute:
111        start_time = datetime.datetime.now()
112        print("Conversion started at:", start_time)
113    with Pool() as pool:
114        pool.starmap(process_file, [(args.execute, file, args.in_dir, args.out_dir, jason, args.formats, args.rules, counter, total_files) for file in files_to_process])
115    if args.execute:
116        end_time = datetime.datetime.now()
117        print("Conversion finished at:", end_time)
118        print("Duration:", end_time - start_time)
119
120if __name__ == '__main__':
121    main()
122

jason_watchdog

New in version 1.1.0.

Monitor a directory for new files and process them with JASON.

Usage:

jason_watchdog [-h] indir outdir --rules RULES [--formats FORMATS [FORMATS ...]]
               [--extensions EXTENSIONS [EXTENSIONS ...]] [--metadata-handler METADATA_HANDLER]

Description:

The jason_watchdog tool monitors a specified input directory (indir) for new data files, processes them automatically using JASON and outputs the results to a specified output directory (outdir). It can be configured to watch for specific file extensions.

By leveraging the --rules option, users can integrate automatic processing, analysis, and layout generation for reporting purposes.

Additionally, the tool can be configured to use a custom metadata handler for extracting and storing metadata from the processed files, allowing for flexible integration with existing data management systems.

Useful for: automation of data conversion pipelines using BeautifulJASON and JASON.

Options:

  • indir: The directory to monitor for new JASON documents.

  • outdir: The directory where processed output files will be saved.

  • --rules: Path to the rules file (e.g., watchdog.jjr) that defines the processing logic.

  • --formats: Output formats for processed files (e.g., pdf, jjh5). Multiple formats can be specified.

  • --extensions: File extensions to monitor (e.g., jdf, jdx). Multiple extensions can be specified.

  • --metadata-handler: Path to a Python module (e.g., dbwriter.py) that implements metadata extraction and storage.

Usage examples:

  1. Monitor a Directory and Process Files:

    To monitor the input directory for .jdf files, apply the rules in watchdog.jjr, and save the output in pdf format to the output directory:

    jason_watchdog ./input ./output --rules watchdog.jjr --formats pdf --extensions jdf

  2. Use a Custom Metadata Handler:

    To monitor the input directory, process .jdf files, and use a custom metadata handler (dbwriter.py) to extract and store metadata:

    jason_watchdog ./input ./output --rules watchdog.jjr --formats pdf --extensions jdf --metadata-handler ./dbwriter.py

batch_extract_integrals

New in version 1.1.0.

Extract integrals and parameters from .jjh5 files into a CSV.

Usage:

batch_extract_integrals [-h] jjh5_path csv_path [-p PARAMETERS ...]

Description:

Iterates over .jjh5 files in a directory and extracts user-specified parameters and multiplet integrals. The result is written into a CSV file with one row per document.

This tool is typically used after the jason_batch_convert tool has been applied with predefined integration rules. The jason_batch_convert tool ensures that the .jjh5 files contain the necessary integrals and metadata for extraction.

Parameters syntax: group/parameter or group/parameter[index], matching the naming conventions in JASON.

Example:

batch_extract_integrals ./jjh5_files output.csv -p parameters/ACTUAL_START_TIME jason_parameters/SpectrometerFrequencies[0]

Useful for: quick extraction of structured integral data and metadata for post-processing and reporting.

Source code:

The source code for this tool is provided for users to review, explore, and gain insights into its implementation. However, the tool itself is fully functional and ready to use as-is.

  1##
  2##-----------------------------------------------------------------------------
  3##
  4## Copyright (c) 2024 JEOL Ltd.
  5## 1-2 Musashino 3-Chome
  6## Akishima Tokyo 196-8558 Japan
  7##
  8## This software is provided under the MIT License. For full license information,
  9## see the LICENSE file in the project root or visit https://opensource.org/licenses/MIT
 10##
 11##++---------------------------------------------------------------------------
 12##
 13## ModuleName : BeautifulJASON
 14## ModuleType : Python API for JASON desktop application and JJH5 documents
 15## Purpose : Automate processing, analysis, and report generation with JASON
 16## Author : Nikolay Larin
 17## Language : Python
 18##
 19####---------------------------------------------------------------------------
 20##
 21
 22import argparse
 23import os
 24import csv
 25import io
 26import beautifuljason as bjason
 27
 28def escape_newlines_for_csv(value):
 29    """Escapes newline characters in a string for CSV compatibility."""
 30    value = str(bjason.utils.ensure_str(value))
 31    # Replace newline characters with literal \n to ensure CSV compatibility
 32    return value.replace('\n', '\\n').replace('\r', '\\r')
 33    
 34def extract_integrals(jjh5_file_path: str, csv_writer: csv.writer, params: list[str]):
 35    """Extracts integrals from a JJH5 file and writes them to a CSV file using csv.writer."""
 36    try:
 37        base_name = os.path.basename(jjh5_file_path)
 38        file_name_without_extension, _ = os.path.splitext(base_name)
 39        
 40        with bjason.Document(jjh5_file_path, mode="r") as doc:
 41            spec = doc.nmr_data[0]
 42            row = [file_name_without_extension]  # Start row with file name
 43            
 44            for param in params:
 45                param_parts = param.split('/')
 46                param_group = param_parts[0]
 47                param_name = param_parts[1]
 48                param_index = None
 49                if '[' in param_name:
 50                    param_name, param_index = param_name.split('[')
 51                    param_index = int(param_index[:-1])
 52
 53                if param_group == "jason_parameters":
 54                    param_value = spec.spec_info.get_param(param_name)
 55                else:
 56                    param_value = spec.raw_data.spec_info.get_orig_param(param_group, param_name)
 57
 58                if param_index is not None and param_value is not None:
 59                    if hasattr(param_value, '__getitem__'):
 60                        param_value = param_value[param_index]
 61
 62                row.append(escape_newlines_for_csv(param_value))
 63                
 64            row.extend(integral.value_hz for integral in spec.multiplets)
 65            csv_writer.writerow(row)
 66    except Exception as e:
 67        print(f"Error processing file '{jjh5_file_path}': {e}")
 68
 69def iterate_jjh5_files(jjh5_path, csv_path, params):
 70    if not os.path.isdir(jjh5_path):
 71        print(f"The directory '{jjh5_path}' does not exist.")
 72        return
 73    if not os.listdir(jjh5_path):
 74        print("The directory is empty.")
 75        return
 76
 77    try:
 78        with open(csv_path, "w", newline='', encoding="utf-8") as csv_file:
 79            csv_writer = csv.writer(csv_file, quoting=csv.QUOTE_MINIMAL)
 80            found_files = False
 81            for file_name in os.listdir(jjh5_path):
 82                if file_name.endswith(".jjh5"):
 83                    found_files = True
 84                    print(f"Processing file: {file_name}")
 85                    extract_integrals(os.path.join(jjh5_path, file_name), csv_writer, params)
 86            if not found_files:
 87                print("No .jjh5 files found in the directory.")
 88    except IOError as e:
 89        print(f"Failed to write to CSV file {csv_path}: {e}")
 90
 91def main():
 92    parser = argparse.ArgumentParser(
 93        description="Iterates through .jjh5 files in a directory and extracts integrals to a CSV file.",
 94        usage="batch_extract_integrals [-h] jjh5_path csv_path [-p PARAMETERS ...]",
 95        epilog="example: batch_extract_integrals path/to/jjh5_files path/to/output.csv -p parameters/ACTUAL_START_TIME jason_parameters/SpectrometerFrequencies[0]"
 96        )
 97    parser.add_argument("jjh5_path", type=str, help="The path to the directory containing .jjh5 files")
 98    parser.add_argument("csv_path", type=str, help="The path to the output CSV file. If the file exists, it will be overwritten.")
 99    parser.add_argument("-p", "--parameters", type=str, nargs='+', help="List parameters to extract, each formatted as 'group/parameter' or 'group/parameter[index]'. Ensure to match the exact case used in the JASON GUI. Example: parameters/ACTUAL_START_TIME jason_parameters/SpectrometerFrequencies[0]")
100    
101    args = parser.parse_args()
102    output_dir = os.path.dirname(args.csv_path)
103    if not output_dir:
104        output_dir = '.'  # If no directory is provided, use the current directory
105    if not os.path.exists(output_dir):
106        print(f"Output directory '{output_dir}' does not exist.")
107        return
108    
109    iterate_jjh5_files(args.jjh5_path, args.csv_path, args.parameters if args.parameters else [])
110
111if __name__ == "__main__":
112    main()

Each tool in the beautifuljason.tools package can be used independently and serves a complementary purpose in automating workflows with JASON and BeautifulJASON. These tools are designed to streamline tasks such as configuration management, batch data conversion, and data extraction, making them versatile and efficient for various use cases.