00 — Verify data provenance and quality
Before creating a metric, we assess:
- the origin of the data and the rights governing their use
- the point in time at which the information was actually available
- stability of coverage over time
- changes in panel or sample composition
- missing values and revisions to historical data
- representativeness and potential selection bias
With panel or alternative data, changes in sample composition can themselves create a false signal. We therefore separate changes in economic behaviour from changes in what the dataset happens to cover at a given time.
01 — Ask a testable question
We define:
- what we want to explain
- which variable we will measure
- what we will compare it with
- the time horizon over which we expect the relationship to exist
- what would falsify the hypothesis
A hypothesis must be formulated so that it can fail.
02 — Structure and measure
We turn raw information into consistent time series and comparable variables.
Depending on the question, we may work with:
- year-over-year or month-over-month change
- relative change versus a peer group
- a standardised deviation from a historical baseline
- a factor or composite indicator
We choose the methodology to fit the data and the research question, not the other way around.
03 — Test the relationship and alternatives
We examine whether the variable is related to the outcome we are trying to explain. Depending on the structure of the data, we use time-series methods, panel models, distributed-lag models or other appropriate methods of applied statistics.
At the same time, we control for relevant alternative explanations, such as industry effects, seasonality, company size or broad market movement.
04 — Test lead time and timing
It is not enough to know that two variables are related. We need to know:
- whether the information is contemporaneous, lagging or leading
- how stable its lead time is
- whether the lead changes across different periods
- when its informational value begins to decay
We do not automatically treat evidence of predictive lead as proof of causality.
05 — Validate robustness out of sample
We test the relationship on data and periods that were not used to develop it.
We use sequential, or walk-forward, testing. Training and test windows are separated to prevent information from leaking between them.
When we test multiple variants and hypotheses, we record how many were tried and adjust for the risk that the best result arose by chance.
06 — Evaluate uncertainty and conditions of validity
The output is not certainty. We assess:
- how strong the evidence is
- how much uncertainty remains
- under which conditions the relationship holds
- when it stops being useful
- how quickly its informational value decays over time
The decay of informational value is part of the result, not something to hide.
07 — Decide the next step
We close each research hypothesis with one of three decisions:
Accept for further use
The relationship held up and has sufficient informational value for further work.
Monitor
The result warrants further monitoring, but the evidence, number of observations or temporal stability is not yet sufficient.
Reject
The hypothesis did not hold, was explained by another factor or lost its practical value.
A negative result is a valid research outcome.
What we consider evidence
Methodology alone is not evidence of a result. We consider an output supported only when the analysis can be traced through:
question → data used → methodology → out-of-sample test → limitations → conclusion
Illustrative examples are used only to explain the process. They are not presented as achieved results or as a track record.
Standard for a published Research Note
We classify a Research Note as a research output only when it is based on a completed analysis and clearly separates:
- the question defined in advance
- the data actually used and when they were available
- the methodology and control variables
- the out-of-sample result
- robustness tests
- limitations and conditions of failure
- the final conclusion, including a negative result where applicable
Until these conditions are met, the material is described as a methodological or validation framework, not as a case study with an achieved result.
Specialised market validation
For hypotheses assessed under financial-market conditions, an additional control layer applies. We examine information timing, feasibility, costs, liquidity, capacity and the behaviour of results under stress scenarios.