Command Line Tools
The beautifuljason.tools package provides a set of ready-to-use command-line tools to support batch processing and configuration tasks for the JASON desktop application and JJH5 documents.
These tools are fully functional, ready to use, and designed to be run directly from the terminal. They allow users to automate workflows, manage configurations, and process data efficiently.
jason_config
Manage JASON installation paths used by BeautifulJASON.
Usage:
jason_config [--display] [--add_path PATH] [--del_path INDEX]
[--set_preferred INDEX] [--reset]
Description:
Allows inspection and modification of the list of known JASON installation paths used by the library. Includes features for setting preferred paths, resetting configuration, and verifying installation validity.
Typical use cases:
Configuring BeautifulJASON to point to a specific JASON installation
Resetting or managing existing JASON paths
Debugging JASON configuration issues
Usage examples:
Display the Current Configuration:
To view the current path settings for the JASON application:
jason_config --display
Add a New JASON Path:
If the JASON application resides in a different location than the detected default paths:
jason_config --add_path /path/to/your/jason/application
Be sure to replace /path/to/your/jason/application with the actual path to your JASON executable.
For Additional Commands and Options:
If you need more details about available commands or want to explore other options:
jason_config --help
Always ensure that the specified path points directly to the JASON executable for BeautifulJASON to function correctly.
jason_batch_convert
Batch convert input files to various formats.
Usage:
jason_batch_convert [-h] in_dir out_dir --formats FORMATS [FORMATS ...]
(--extensions EXTENSIONS [EXTENSIONS ...] | --patterns PATTERNS [PATTERNS ...])
[--rules RULES] [--execute]
Description:
Recursively scans the input directory for files matching specified extensions or path patterns and converts them to the selected formats. The tool can operate in dry mode (default) to list the matching files or in execution mode using the --execute option to perform the actual conversion.
By leveraging the --rules option, users can integrate automatic processing, analysis, and layout generation for reporting purposes. This makes the tool an essential first step in preparing data for advanced analysis workflows.
Formats supported: jjh5, jjj, jdx, jdf, pdf, png, jpg, svg
Useful for: automation of data conversion pipelines using BeautifulJASON and JASON.
Usage examples:
Dry Run to List Matching Files:
To scan the input directory for .jdf files and list the matching files without performing any conversion:
jason_batch_convert ./input ./output --formats pdf --extensions jdf
Convert Files to Multiple Formats:
To convert all .jdf files in the input directory to both pdf and jjh5 formats:
jason_batch_convert ./input ./output --formats pdf jjh5 --extensions jdf --execute
Use Path Patterns for File Selection:
To convert files matching a specific pattern (e.g., files starting with sample_) to pdf format:
jason_batch_convert ./input ./output --formats pdf --patterns sample_* --execute
Convert Files in Specific Directory Structures:
To process datasets organized in specific directory structures, such as fid files located in subdirectories named 10, and convert them to jjh5 format:
jason_batch_convert ./input ./output --formats jjh5 --patterns */10/fid --execute
This example matches fid files located in subdirectories named 10 within the input directory. The */10/fid pattern ensures that the tool looks for fid files specifically in directories named 10, regardless of their parent directory. The */ part acts as a wildcard, allowing the tool to search through all subdirectories of the input directory to find matching paths.
Apply Custom Rules During Conversion:
To apply a specific set of processing rules during conversion:
jason_batch_convert ./input ./output --formats pdf --extensions jdf --rules custom_rules.jjr --execute
Source code:
The source code for this tool is provided for users to review, explore, and gain insights into its implementation. However, the tool itself is fully functional and ready to use as-is.
1##
2##-----------------------------------------------------------------------------
3##
4## Copyright (c) 2024 JEOL Ltd.
5## 1-2 Musashino 3-Chome
6## Akishima Tokyo 196-8558 Japan
7##
8## This software is provided under the MIT License. For full license information,
9## see the LICENSE file in the project root or visit https://opensource.org/licenses/MIT
10##
11##++---------------------------------------------------------------------------
12##
13## ModuleName : BeautifulJASON
14## ModuleType : Python API for JASON desktop application and JJH5 documents
15## Purpose : Automate processing, analysis, and report generation with JASON
16## Author : Nikolay Larin
17## Language : Python
18##
19####---------------------------------------------------------------------------
20##
21
22import os
23import datetime
24from multiprocessing import Pool, Manager
25from multiprocessing.managers import ValueProxy
26from pathlib import Path
27import argparse
28import glob
29from colorama import Fore, init
30
31import beautifuljason as bjason
32
33def unique_output_filename(file, extension):
34 """Generate a unique output filename based on the input file path and desired extension."""
35 unique_base_name = file.replace(os.sep, '_') # Replace path separators with underscores
36 return unique_base_name + '.' + extension
37
38def process_file(execute: bool, file: str, in_dir: str, out_dir: str, jason: bjason.JASON, formats: list[str], rules: str, counter: ValueProxy, total_files: int):
39 """Process a single file, converting it to specified formats."""
40 out_fnames = [unique_output_filename(file, format) for format in formats]
41 if execute:
42 global processed_files
43 with jason.create_document(os.path.join(in_dir, file), rules=rules) as doc:
44 doc.close()
45 jason.save(doc, [os.path.join(out_dir, fname) for fname in out_fnames])
46 counter.value += 1
47 print(f"{counter.value}/{total_files}: {file} => {', '.join(out_fnames)}")
48
49def main():
50 init(autoreset=True) # Initialize colorama
51
52 parser = argparse.ArgumentParser(
53 description='Convert files from a specified directory based on extensions or patterns.',
54 usage='jason_batch_convert [-h] in_dir out_dir --formats FORMATS [FORMATS...] (--extensions EXTENSIONS [EXTENSIONS ...] | --patterns PATTERNS [PATTERNS ...]) [--rules RULES] [--execute]'
55 )
56 parser.add_argument('in_dir', help='Root directory containing files to be converted.')
57 parser.add_argument('out_dir', help='Directory where the converted files will be saved.')
58 parser.add_argument('--formats', nargs='+', required=True,
59 choices=['jjh5', 'jjj', 'jdx', 'jdf', 'pdf', 'png', 'jpg', 'svg'],
60 help='List of desired output file formats. Multiple formats can be specified.')
61 parser.add_argument('--rules', type=str, default="off",
62 help='Path to the rules file or rule library name.')
63 parser.add_argument('--execute', action='store_true',
64 help='Execute the file conversions. If not specified, will perform a dry run, listing the files to be converted. It is recommended to run a dry run first.')
65
66 group = parser.add_mutually_exclusive_group(required=True)
67 group.add_argument('--extensions', nargs='+', help='List of file extensions to convert, e.g., jdf, als, jdx, mol, etc.')
68 group.add_argument('--patterns', nargs='+', help='List of file path patterns to convert, e.g., */*.fid/fid, */10/fid, etc. Patterns cannot end with "/*".')
69
70 args = parser.parse_args()
71
72 # Convert in_dir and out_dir to absolute paths
73 args.in_dir = os.path.abspath(args.in_dir)
74 args.out_dir = os.path.abspath(args.out_dir)
75
76 if not args.execute:
77 print(f"{Fore.YELLOW}Dry run mode! No files will be converted.")
78 print("Input directory:", args.in_dir)
79 print("Output directory:", args.out_dir)
80
81 # Initialize Jason
82 jason = bjason.JASON()
83
84 # Create output directory
85 Path(args.out_dir).mkdir(parents=True, exist_ok=True)
86
87 # Find and process files matching patterns or extensions
88 print("Searching for files...")
89 files_to_process = []
90 if args.patterns:
91 for pattern in args.patterns:
92 if pattern.endswith("/*"):
93 raise ValueError("Invalid pattern. Patterns cannot end with '/*'.")
94 pattern_files = glob.glob(os.path.join(args.in_dir, pattern), recursive=True)
95 files_to_process.extend([os.path.relpath(f, start=args.in_dir) for f in pattern_files if os.path.isfile(f)])
96 else:
97 for root, dirs, files in os.walk(args.in_dir):
98 for file in files:
99 if file.split('.')[-1] in args.extensions:
100 full_path = os.path.join(root, file)
101 if os.path.isfile(full_path):
102 files_to_process.append(os.path.relpath(full_path, start=args.in_dir))
103
104 total_files = len(files_to_process)
105 print(f"{total_files} files found in the input directory.")
106
107 manager = Manager()
108 counter = manager.Value('i', 0) # integer counter initialized to 0
109
110 if args.execute:
111 start_time = datetime.datetime.now()
112 print("Conversion started at:", start_time)
113 with Pool() as pool:
114 pool.starmap(process_file, [(args.execute, file, args.in_dir, args.out_dir, jason, args.formats, args.rules, counter, total_files) for file in files_to_process])
115 if args.execute:
116 end_time = datetime.datetime.now()
117 print("Conversion finished at:", end_time)
118 print("Duration:", end_time - start_time)
119
120if __name__ == '__main__':
121 main()
122
jason_watchdog
New in version 1.1.0.
Monitor a directory for new files and process them with JASON.
Usage:
jason_watchdog [-h] indir outdir --rules RULES [--formats FORMATS [FORMATS ...]]
[--extensions EXTENSIONS [EXTENSIONS ...]] [--metadata-handler METADATA_HANDLER]
Description:
The jason_watchdog tool monitors a specified input directory (indir) for new data files, processes them automatically using JASON and outputs the results to a specified output directory (outdir). It can be configured to watch for specific file extensions.
By leveraging the --rules option, users can integrate automatic processing, analysis, and layout generation for reporting purposes.
Additionally, the tool can be configured to use a custom metadata handler for extracting and storing metadata from the processed files, allowing for flexible integration with existing data management systems.
Useful for: automation of data conversion pipelines using BeautifulJASON and JASON.
Options:
indir: The directory to monitor for new JASON documents.
outdir: The directory where processed output files will be saved.
--rules: Path to the rules file (e.g., watchdog.jjr) that defines the processing logic.
--formats: Output formats for processed files (e.g., pdf, jjh5). Multiple formats can be specified.
--extensions: File extensions to monitor (e.g., jdf, jdx). Multiple extensions can be specified.
--metadata-handler: Path to a Python module (e.g., dbwriter.py) that implements metadata extraction and storage.
Usage examples:
Monitor a Directory and Process Files:
To monitor the input directory for .jdf files, apply the rules in watchdog.jjr, and save the output in pdf format to the output directory:
jason_watchdog ./input ./output --rules watchdog.jjr --formats pdf --extensions jdf
Use a Custom Metadata Handler:
To monitor the input directory, process .jdf files, and use a custom metadata handler (dbwriter.py) to extract and store metadata:
jason_watchdog ./input ./output --rules watchdog.jjr --formats pdf --extensions jdf --metadata-handler ./dbwriter.py
batch_extract_integrals
New in version 1.1.0.
Extract integrals and parameters from .jjh5 files into a CSV.
Usage:
batch_extract_integrals [-h] jjh5_path csv_path [-p PARAMETERS ...]
Description:
Iterates over .jjh5 files in a directory and extracts user-specified parameters and multiplet integrals. The result is written into a CSV file with one row per document.
This tool is typically used after the jason_batch_convert tool has been applied with predefined integration rules. The jason_batch_convert tool ensures that the .jjh5 files contain the necessary integrals and metadata for extraction.
Parameters syntax: group/parameter or group/parameter[index], matching the naming conventions in JASON.
Example:
batch_extract_integrals ./jjh5_files output.csv -p parameters/ACTUAL_START_TIME jason_parameters/SpectrometerFrequencies[0]
Useful for: quick extraction of structured integral data and metadata for post-processing and reporting.
Source code:
The source code for this tool is provided for users to review, explore, and gain insights into its implementation. However, the tool itself is fully functional and ready to use as-is.
1##
2##-----------------------------------------------------------------------------
3##
4## Copyright (c) 2024 JEOL Ltd.
5## 1-2 Musashino 3-Chome
6## Akishima Tokyo 196-8558 Japan
7##
8## This software is provided under the MIT License. For full license information,
9## see the LICENSE file in the project root or visit https://opensource.org/licenses/MIT
10##
11##++---------------------------------------------------------------------------
12##
13## ModuleName : BeautifulJASON
14## ModuleType : Python API for JASON desktop application and JJH5 documents
15## Purpose : Automate processing, analysis, and report generation with JASON
16## Author : Nikolay Larin
17## Language : Python
18##
19####---------------------------------------------------------------------------
20##
21
22import argparse
23import os
24import csv
25import io
26import beautifuljason as bjason
27
28def escape_newlines_for_csv(value):
29 """Escapes newline characters in a string for CSV compatibility."""
30 value = str(bjason.utils.ensure_str(value))
31 # Replace newline characters with literal \n to ensure CSV compatibility
32 return value.replace('\n', '\\n').replace('\r', '\\r')
33
34def extract_integrals(jjh5_file_path: str, csv_writer: csv.writer, params: list[str]):
35 """Extracts integrals from a JJH5 file and writes them to a CSV file using csv.writer."""
36 try:
37 base_name = os.path.basename(jjh5_file_path)
38 file_name_without_extension, _ = os.path.splitext(base_name)
39
40 with bjason.Document(jjh5_file_path, mode="r") as doc:
41 spec = doc.nmr_data[0]
42 row = [file_name_without_extension] # Start row with file name
43
44 for param in params:
45 param_parts = param.split('/')
46 param_group = param_parts[0]
47 param_name = param_parts[1]
48 param_index = None
49 if '[' in param_name:
50 param_name, param_index = param_name.split('[')
51 param_index = int(param_index[:-1])
52
53 if param_group == "jason_parameters":
54 param_value = spec.spec_info.get_param(param_name)
55 else:
56 param_value = spec.raw_data.spec_info.get_orig_param(param_group, param_name)
57
58 if param_index is not None and param_value is not None:
59 if hasattr(param_value, '__getitem__'):
60 param_value = param_value[param_index]
61
62 row.append(escape_newlines_for_csv(param_value))
63
64 row.extend(integral.value_hz for integral in spec.multiplets)
65 csv_writer.writerow(row)
66 except Exception as e:
67 print(f"Error processing file '{jjh5_file_path}': {e}")
68
69def iterate_jjh5_files(jjh5_path, csv_path, params):
70 if not os.path.isdir(jjh5_path):
71 print(f"The directory '{jjh5_path}' does not exist.")
72 return
73 if not os.listdir(jjh5_path):
74 print("The directory is empty.")
75 return
76
77 try:
78 with open(csv_path, "w", newline='', encoding="utf-8") as csv_file:
79 csv_writer = csv.writer(csv_file, quoting=csv.QUOTE_MINIMAL)
80 found_files = False
81 for file_name in os.listdir(jjh5_path):
82 if file_name.endswith(".jjh5"):
83 found_files = True
84 print(f"Processing file: {file_name}")
85 extract_integrals(os.path.join(jjh5_path, file_name), csv_writer, params)
86 if not found_files:
87 print("No .jjh5 files found in the directory.")
88 except IOError as e:
89 print(f"Failed to write to CSV file {csv_path}: {e}")
90
91def main():
92 parser = argparse.ArgumentParser(
93 description="Iterates through .jjh5 files in a directory and extracts integrals to a CSV file.",
94 usage="batch_extract_integrals [-h] jjh5_path csv_path [-p PARAMETERS ...]",
95 epilog="example: batch_extract_integrals path/to/jjh5_files path/to/output.csv -p parameters/ACTUAL_START_TIME jason_parameters/SpectrometerFrequencies[0]"
96 )
97 parser.add_argument("jjh5_path", type=str, help="The path to the directory containing .jjh5 files")
98 parser.add_argument("csv_path", type=str, help="The path to the output CSV file. If the file exists, it will be overwritten.")
99 parser.add_argument("-p", "--parameters", type=str, nargs='+', help="List parameters to extract, each formatted as 'group/parameter' or 'group/parameter[index]'. Ensure to match the exact case used in the JASON GUI. Example: parameters/ACTUAL_START_TIME jason_parameters/SpectrometerFrequencies[0]")
100
101 args = parser.parse_args()
102 output_dir = os.path.dirname(args.csv_path)
103 if not output_dir:
104 output_dir = '.' # If no directory is provided, use the current directory
105 if not os.path.exists(output_dir):
106 print(f"Output directory '{output_dir}' does not exist.")
107 return
108
109 iterate_jjh5_files(args.jjh5_path, args.csv_path, args.parameters if args.parameters else [])
110
111if __name__ == "__main__":
112 main()
Each tool in the beautifuljason.tools package can be used independently and serves a complementary purpose in automating workflows with JASON and BeautifulJASON. These tools are designed to streamline tasks such as configuration management, batch data conversion, and data extraction, making them versatile and efficient for various use cases.