A benchmarking tool for linear codes

[sagetrac-dlucas]

This ticket proposes a tool to easily run experiments on linear codes (encoding, decoding...), gather data and process it.

CC: @johanrosenkilde @ClementPernet

Component: coding theory

Branch/Commit: u/dlucas/benchmarking_tool @ fe0b956

Issue created by migration from https://trac.sagemath.org/ticket/20526

[sagetrac-dlucas]

Branch: u/dlucas/benchmarking_tool

[sagetrac-dlucas]

Commit: 5e18f8c

[sagetrac-dlucas]

comment:2

I pushed the first version. Some minor issues still exist, I'll fix these in the next few days.

---

New commits:

cd86b18	First version of a benchmarking tool for linear codes
5e18f8c	Added ticket number to experimental flag

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

8726309

Changed how empty benchmarks are managed in __add__. Tweaks wrt. experimental_data

[sagetrac-git]

Changed commit from 5e18f8c to 8726309

[sagetrac-dlucas]

comment:4

Hello,

I made some changes:

• I removed the code, the encoder, the decoder and the channel from the dictionary experimental_data, as this never changes through the tests for a given benchmark.
• I strengthened how empty benchmarks are managed in addition. Consider B1 and B2 two benchmarks. If:
  • B1 and B2 are empty, an exception is returned,
  • B1 or B2 are empty, the empty benchmark is ignored.

This is still under new state as there's some remaining bugs (e.g. "experimental" warning is called everytime a Benchmark is constructed, even inside the same session).

[sagetrac-git]

Changed commit from 8726309 to 0ab93ec

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

5dafb07	Removed encoder from constructor's input and made related adjustments in doc and methods
18732e1	Adding two empty benchmarks now returns an empty benchmark
536ed61	Added an extra message to `_repr_` in case of an empty benchmark, enchanced docstring for _repr_
5030919	Updated to 7.2rc1
32dd3d3	no_tests is now a class argument, added related getter
0dfdbf0	Replaced exception by a return statement in run() on empty benchmarks
7212b33	Added a distinction between decoding_error and decoding_failure, refined decoding_success_rate accordingly
bc29f26	Refined experimental_data method, which now allows request for specific data
a82a477	Fixed syntax errors in documentation
0ab93ec	Changes and fixes to docstrings and doctests

[sagetrac-dlucas]

comment:7

Hello,

Here's a change log for this ticket:

• I removed encoder from this class. One cannot pass an encoder to a benchmark anymore, and all related methods have been deleted. Instead, I replaced the old call Benchmark(E, D, Chan) by Benchmark(C, D, Chan).

• Adding two empty benchmarks now returns an empty benchmark and no longer an exception.

• I added better messages in case one tries to manipulate an empty benchmark: for instance. calling Benchmark.run() now returns a proper message which explains that nothing will be done.

• I refined the notion of "success rate". It now considers decoding errors (when the decoder fails at returning anything) and decoding failures (when the decoder returns a codeword which is not the expected one). I also enhanced decoding_success_rate, which now allows the user to select between success rate wrt. failures or errors.

• I globally improved the documentation.

One problem remains: the experimental warning triggered by the decorator @experimental activates every time one creates a benchmark, which is annoying and also breaks all my doctests.
This is quite weird as the documentation for this decorator says it will only be triggered once/session.
Still investigating on this...

David

[johanrosenkilde]

comment:8

Sounds good! Quick remark (didn't check the code yet): In "A Course In Error-Correcting Codes" by Justesen and Høholdt then "decoding error" is defined as the decoder returning a codeword which was not the intended, and "decoding failure" is defined as the decoder not returning the wanted codeword (i.e. either returning nothing or returning a wrong one).

That is, P(failure) >= P(error) and P(failure) - P(error) = P(not returning anything).

Perhaps this implementation should reflect this (non-universally adopted) convention.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

d2d6e26

Improved data selection in experimental_data method