NEWS.md
read_opus(dsn, data_only = TRUE). Previously, extraction of the timestamp failed and data extracted errored in that case, because there was the required "history" and "sample" blocks weren’t extracted temporarily before, as required. Now, read_opus(dsn, data_only = TRUE) successfully extracts an extra element basic_metadata, as it does for data_only = FALSE (the default). This extra information does unlikely to break existing pipeline code that extracts spectra with {opusreader2}, because it is a nested list element. This patch release also resolves a warning when parsing time information, that was due to an extra tab ("\t") that was present in the history text for specific files. Thanks @mtalluto for the fix. Added extra tests to check for errors and warnings in the example files for both data_only = FALSE and data_only = TRUE). Thanks to @dylanbeaudette and @esteveze for reporting the failing extraction of metadata. Issue report: #104. PR fixed: #105.@export tags were removed and @internal added for calc_parameter_chunk_size(), which made those functions unavailable even internally ("Error in UseMethod("calc_parameter_chunk_size") : no applicable method for 'calc_parameter_chunk_size' applied to an object of class "parameter")"
read_opus(): in return element ab, state Log10 explicitly for calculating apparent absorbance (#94; @zecoljls).read_opus(), read_opus_single()
calc_parameter_chunk_size()
opus_file()
./inst/extdata/new_data/issue81_A1.1.0. read_opus() returns this block as "quality_test_report" in the list output."unknown" elements in the output of the read_opus() list../inst/extdata/new_data/issue94_RT_01_1_23-02-21_13-23-54.0: from Bruker 2023 Alpha II mid-IR spectrometer. Due to internal refactoring of header parsing (see below) (#94)./inst/extdata/new_data/issue82_Opus_test: from Bruker MPA FT-IR spectrometer. Parse block "b0-c0-t144-a1", text type 144 with special offset in parse_chunk.parameter(). For now classify this block as block type "report_unknown" (waiting finalize naming until confirmed with screenshots from the Bruker OPUS sofware). Also fix time_saved by not relying on language settings (#82)parse_header().raw vectors instead of connection objects to read binary data. Parse raw vectors directly for functions in read_bin_types() and use subsetting to slice raw vectors in base::readBin() calls instead instead of seek(), which was used previously to reposition cursors in raw connections.get_meta_timestamp(): omit language dependent logic using "time saved" regular expressions for matching time saved from history block. The first time of sorted POSIXct candidates will be returned as time saved.basic_metadata list element for “opusreader2” class containing key metadata (#85)Name first level of list (class "list_opusreader2") with base file name of given data source name (DSN) (#83)
Fix "list_opusreader2" indenting when reading files in parallel (#80)
Add support for progress bars in read_opus() (#75)
Introduce type-stable classes for read_opus() and read_opus_single() output (#72):
read_opus(..., parallel = TRUE): unlist resulting list one level (chunk level); #80.read_opus() when reading multiple files in parallel #75.read_opus(): Read one or more OPUS files from data source name (dsn)read_opus_single(): Read a single OPUS filecalc_parameter_chunk_size(): Calculate the parameter chunk size in bytesread_opus(): S3 class c("list_opusreader2", "list")
read_opus_single(): S3 class c("opusreader2", "list")
Internal refactoring (R/create_dataset.R). Implement a new key-value mapping logic for assigning the integer coded header information. The new order in the (composite) key strings follows the sequence of block, channel, text and additional type information. The better line-by-line layout of composite keys and mapped information types simplifies the detection of new kind of spectral data and parameters that are encoded in header entries (#60).
Introduce consistent and proactive error reporting when a composite key in are not yet mapped because they are not yet known (R/create_dataset.R). This error message includes a recipe how to report new OPUS files with yet unsupported block types (i.e. new instrument features) for {opusreader2}. Together with the composite key generated from the respective the header entry, a step-by-step reporting as GitHub issue is proposed. (#60)
parse_opus()
Start versioning with {fledge}.
spectral-cockpit.com proudly introduces {opusreader2} to read binary files from FT-IR devices from Bruker Optics GmbH & Co in R. It is a powerhouse that fuels speedy extract-transform-load (ETL) data pipelines in spectroscopy applications. You can continue using state-of-the-art commercial devices for what they are good at: measurements. Meanwhile, you can rely on open source technology and trans-disciplinary knowledge to design data processes, and make best use of the spectroscopic source of information.
{opusreader2} parses and decodes the at first glance puzzling file header first. The implementation then uses this mapped information as a recipe to read particular data types from different blocks. Specific byte chunks to be interpreted are defined by position (offset), read length, bytes per element, and type (e.g., string, float). With this, all the data can be read and parsed. We mitigate lock-in at file level. Hence we foster reproducible and trustworthy processes in spectral workflows. Nowadays, the new business logic is being more and more transparent in code, methods used and services offered. Tightly link and make input data, metadata and outcomes available for economical scaling-up of diagnostics.
Providing the data and metadata from measurements connects downstream tasks in order to make IR spectroscopy a ready-made, automatec for diagnostics and monitoring (platform):
With our package you can directly read and parse from binary files without compromising a single bit of precious information saved in these filled OPUS binary files.
read_opus() is the main function exposed that reads and parses OPUS binary files from various data sources names (dsn). Currently, we support the following dsn types:
.<integer> extension (Usually starting from .0 for unique sample names per measurement.File names of OPUS files can possibly include plate positions that are postfixed to the sample names. This is an option in OPUSLab. Kindly note that the associated metadata (sample name/ID) and plate position are also stored internally so that file name changes after measurement could be tracked.
read_opus offers four arguments:
dsn: data source namedata_only: switch to extract only spectral data blocks without additional information like measurement parameters or environmental conditions.parallel: not enabled by default. Speed up reads of 1000s of files by chunking list of files across parallel workers. Cross-platform via unified {future} framework in R.progress_bar: optionally show interactive progress bar for single-threaded or asynchronous reads.The interface is minimal and the job of the generic reader function is well defined by design. This is to make maintenance easy and to avoid breaking changes in future releases of the package. We importantly avoid feature overload like this. We plan to release specific helper and wrapper functions that can come in handy for tailored uses and diagnostic environments. They may also extract or post-process spectroscopic data and metadata pipelines. Check out more soon in future releases.