Problem ordering, topics covered, and understanding of concepts required

[kytrinyx]

For the past three years, the ordering of exercises has been done based on gut feelings and wild guesses.

Over time this has proven to work OK-ish, but it's not great. There are easy exercises that get placed too far back, exercises that are too similar to one another, and exercises that are too difficult that end up early in the track, and we mostly don't notice until someone says out loud that they're struggling with something.

@IanWhitney did a thorough analysis of the rust track in exercism/rust#127, which resulted in reordering the exercises. He wrote about the experience in an essay called Exercism Shouldn't Make You Cry

We're also talking about similar issues in F# (exercism/fsharp#133 - @ErikSchierboom) and Elixir (exercism/elixir#190 - @devonestes), and I'm sure the topic has been mentioned elsewhere and I've missed it.

Back in the day, Peter Minten talked about how we might classify exercises more systematically, in exercism/problem-specifications#63 and exercism/problem-specifications#72.

I think that if we have language-specific classifications and topics, we should do it in the language-specific repository (keeping all the language-specific stuff together).

What if we did this in config.json? We could have a new key, exercises, which contained an array of objects with the problem slug and topics. x-api could be changed to look preferentially at the new key, and fall back to the old one if it's missing. Then we could migrate all the tracks without having to do everything all at once.

The topics would be optional, but having it in the actual codebase would probably help crowdsource this data.

@exercism/track-maintainers Thoughts?

[markijbema]

I think it's a great idea. I think it would be worthwhile to think about a way to make it non-optional though. If you don't, you run the risk of making it like comments in code. If you expose it to users you have to get it right; you could even do this like saying your skill level for a certain thing is at a certain level. Of course, the flipside there is that you then have to handle perception of users.

Conceptually I think it's an interesting question as well whether you should actually enforce a path, or let the user choose to follow a path of their own, with possible other routes (like for instance DuoLingo does). I think the first few exercises you want to let people adhere to a specific order, but after that, it matters less I think.

But all this, if at all interesting, does not need to be the first version of course :)
(also: disclaimer, I haven't been on exercism for quite a long time)

[ErikSchierboom]

I for one think this is a great idea! As a track maintainer, I often struggle with the relative ordering of exercises. Having a consistent way to classify and categorize exercises would greatly help with that. I would suggest that for each exercise, we add the following information:

1. Topics: a list of the topics covered by the exercise. This includes things like control flow, recursion, parsing, etc. Maybe we can come up with a list of default topics that cover 90% of the topics, and let the rest be language-specific.
2. Difficulty: although the topics can help in judging the difficulty of an exercise (parsing is harder than basic control flow, more topics are harder than fewer topics), I think we should also be able to explicitly specify the difficulty. The main reason for this is that although exercises might have similar topics, they can be of vastly different difficulty.

As @choiaa suggested in this issue, we might even be able to automatically update the order of the exercises based on the topics and difficulty.

In the discussion started by @pminten, he suggests to have two also categorize the exercises in either:

• Focus
• Practice
• Challenge

Although I see what he's trying to achieve, I think the same can be accomplished using the suggested difficulty classification.

He also suggested that we specify the topics in the exercise data in the x-common repository, but I think that would not be very convenient and the same exercises might have completely different topics in different languages.

I am also favor of the suggestion of @markijbema that this information should be required in the new format. His point about having a default path of being able to choose a path is also a good one, but I think that's a different discussion.

Overall, I think it's a great idea and I'll gladly help.

[devonestes]

I think it's a great idea, too, but I don't yet know how we'd solve the issue with different exercises being different levels of difficulty in different languages.

For example, the anagram exercise is on the easier side in Elixir or Ruby, but apparently in Rust it's very difficult. At the very least, maybe in the config.json when we list the exercises, we could also list a "relative difficulty" for that exercise as it's implemented in a given language? That way when new exercises are added then the person adding that exercise will have at least a general idea about where to slide it in. Maybe something like this?:

"problems": [
  {
    "_difficulty": 1,
    "name": "hello-world"
  },
  {
    "_difficulty": 3,
    "name": "anagram"
  },
  {
    "_difficulty": 10,
    "name": "forth"
  }
]

Maybe to get better measures of difficulty we could get input from users after they submit an iteration on how hard they felt the problem was in a given language, and that data could be exposed somehow for maintainers to use? That way they'd have some more concrete information on which to base the ordering of exercises.

[masters3d]

I don't have time to dive deep but I am envisioning a solution that is similar to operator precedence. Perhaps we can group each problem and then just worry about the grouping or "concept" ordering and not specifically about each problem order.

[kytrinyx]

I think it's a great idea, too, but I don't yet know how we'd solve the issue with different exercises being different levels of difficulty in different languages.

With the suggested approach, each language track's classification would be independent of other languages. So anagram in Ruby could be classified as easy, and in Rust it could be difficult.

[ErikSchierboom]

And I think that is the only way it will ever make sense. I'm really excited about this possible change!

[kotp]

Is there a list of (generic) programming concepts that could be used and weighted? The list being shared (mostly common) between tracks, but the weights given to the concept specific to those tracks?

[ErikSchierboom]

@kotp I don't know if there is such a list, but we could also try to gather it manually. Here is a quick attempt, in which I have tried to group related concepts:

Basic concepts

• Abstract classes
• Callbacks
• Control-flow (if-else statements)
• Control-flow (loops)
• Events
• Exception handling
• Generics
• Inheritance
• Interfaces
• Pattern matching
• Polymorphism
• Recursion
• Threading
• Type conversion
• Variables

Data types

• Arrays
• Booleans
• Classes
• Dates
• Discriminated unions
• Enumerations
• Floating-point numbers
• Integers
• Lists
• Maps
• Optional values
• Queues
• Records
• Sequences
• Sets
• Stacks
• Strings
• Structs
• Time
• Trees
• Tuples

Problem areas

• Algorithms
• Files
• Filtering
• Games
• Globalization
• Logic
• Networking
• Parallellism
• Parsing
• Pattern recognition
• Performance
• Randomness
• Refactoring
• Searching
• Security
• Sorting
• Text formatting
• Transforming

This is by no means a definitive list, it's just something I compiled myself. Maybe we can use this as a starting point for our discussion?

[kotp]

Definitely like the list. I had started one yesterday, but did not get far.

[rbasso]

The general programming concepts are never so general as they appear. 😄

I think It's nice to have a list of suggested categories, unless the tracks are forced to fit in them.

Maybe it would be more interesting to leave the topics completely open and see what would emerge from the tracks...

[markijbema]

I think the topics will need to differ per track as well, at least, you don't want to restrict them for all languages.

[ErikSchierboom]

@rbasso I don't think we should leave the topics completely open, that will probably lead to a lot of unwanted duplication where similar concepts are named differently in different tracks. We would then have to do cleanup later.

@markijbema The topics will definitely differ per track, but as I said, this list can be something of a starting list. It is not meant to be comprehensive, just helpful :)

[rbasso]

You convinced me, @ErikSchierboom. 😄

I don't think we should leave the topics completely open, that will probably lead to a lot of unwanted duplication where similar concepts are named differently in different tracks. We would then have to do cleanup later.

[ErikSchierboom]

What scale should we use to grade the difficulty? 1 to 10? 1 to 3? Or text: easy, average, hard?

[kotp]

I think it is easier to consider difficulties in grades of 10 or even 0 to 9. Something that is built in and primitive in the language may be 0 or 1, depending on the conceptual difficulty. It also makes it simple to add the grades to make a weight for for an individual exercise based on adding those values.

The report could be easy, average or hard, derived from the numbers. There could be an additional weight that doesn't correspond from the categories directly, but from the reported response from the users that we get in feedback, that can help to fine tune it to where it is perceived to belong as well, over time.