[SOLVED] TSC on Atom CPU and downscaling CPU frequency

[agent_kith]

Thanks again!

Unfortunately it doesnt load

Expanding kernel file...

*** ERROR *** Invalid Snakeoil OS kernel 

Working fine here. Maybe try and download from another browser, and try again?

[frednork]

Thanks again!

Unfortunately it doesnt load

Expanding kernel file...

*** ERROR *** Invalid Snakeoil OS kernel 

Working fine here. Maybe try and download from another browser, and try again?

 

Many Thanks!

working now

[agent_kith]

Many Thanks!

working now  

Cool. With this kernel and the NF9C hardware. Make sure to use cpuset in tandem and isolate all the playing software into just a single kernel. The drift across clocks is pretty bug and I'm think it's a hardware or BIOS configuration issue. Isolating every music software into 1 CPU should alleviate that.

Let me know how it sounds.

[frednork]

Many Thanks!

working now  

Cool. With this kernel and the NF9C hardware. Make sure to use cpuset in tandem and isolate all the playing software into just a single kernel. The drift across clocks is pretty bug and I'm think it's a hardware or BIOS configuration issue. Isolating every music software into 1 CPU should alleviate that.

Let me know how it sounds.  

Hi I have been using Roonbridge and not sure this gets isolated to cpu1.
Finally had a good listen to the 300hz kernel with both tsc and hpet clock and found I preferred the standard kernel in the end. I also compared it with an SOTM SMS200 ultra and really couldnt pick much between them so congrats on a great bit of software.
I also found that reducing vm stat polling "sysctl -w vm.stat_interval=3600" and to a lesser extent setting the hpet timer frequency " echo "8192" | sudo tee --append /proc/sys/dev/hpet/max-user-freq" helped. These were not my ideas but from @rmpfyf on SNA. 

I also tried to lower cpu frequency but the kernel did not allow it, is this something you have tried previously?

[agent_kith]

Hi I have been using Roonbridge and not sure this gets isolated to cpu1.

Not entirely sure as I havn't tested RoonBridge myself. One quick way to test is to go to the Snakeoil menu after RoonBridge has started and see if it's isolated. If it's not there, then it's likely the command to isolation the cores is executed before RoonBridge has started.

I think this happens with MediaCenter, and it's very likely Roon is affected as well (As RoonBridge uses C#, which is like JAVA and have more overheads, hence loading up will be slow).
 

Finally had a good listen to the 300hz kernel with both tsc and hpet clock and found I preferred the standard kernel in the end. I also compared it with an SOTM SMS200 ultra and really couldnt pick much between them so congrats on a great bit of software.

This is because I spent more time tweaking the standard kernel. If I start with a 300 Hz (or some other frequency, say 1000 Hz), then slowly tweak the other stuffs, who knows, it may beat the standard kernel then? Time is the enemy here as there're quite a few things to experiment with. And evaluation can be difficult. e.g. one may prefer the standard kernel, but this may be because it's less revealing than the 300 Hz one. Sometimes it's really hard to make the call. Sometimes to make the difference easier to perceive, I have to dial up the volume a lot higher than I'm comfortable with. And the longer one spend time auditioning the kernels, the more they all begin to sound the same.

It is amazing (and mysterious) why software can do this.
 

I also found that reducing vm stat polling "sysctl -w vm.stat_interval=3600" and to a lesser extent setting the hpet timer frequency " echo "8192" | sudo tee --append /proc/sys/dev/hpet/max-user-freq" helped. These were not my ideas but from @rmpfyf on SNA. 

Cool. I'm thinking of adding a feature that allows folks to enter custom bootup commands. When that is ready you can put these commands there (I'd probably make these the default).

 

I also tried to lower cpu frequency but the kernel did not allow it, is this something you have tried previously? 

CPU Frequency cannot be adjusted with the two kernels you have tried because that part of the code is disabled. Power saving features usually add latencies. You can install the CPUFfreq kernel to add this support if you like (e.g. useful when running on a notebook that's battery powered)...

Some motherboards allow you to adjust the CPU speed by changing the multiplier and/or the bus speed. I remember adjusting the CPU frequency to make it divisible by 44100 before back when I was using a full ATX PC before. FWIW now using a JS-2 with big hashimoto tranny and choke I don't think that's necessary any more. So in a way there may be a relationship between CPU speed and input power. (Disclaimer: I'm only guessing here!).

[agent_kith]

@frednork ,

two quick comments.

Cyclic Test
What's your cyclictest latency numbers? Bear in mind Snakeoil uses nanoseconds (almost everybody else uses microseconds). So while it seem like your values look skyhigh, you need to divide by 1000. The reason to use nanoseconds is it's not difficult to get the latency down to 1 us with the right hardware when using Snakeoil OS. I did this with my Core2Duo a while back, and I still prefer my atom board, despite the worse latency numbers. For all intents and purposes 1 us latency vs 100 us isn't much in my point of view.

More information here. Also not sure if it's mentioned in this thread or somewhere else, the way cyclictest is running at the moment can be a bit deceptive. It is not showing you everything.

cyclicscope is a better presentation and more informative. This is something I intend to work on and hopefully get it out soon. In case you are unaware what that is, it looks something like this:
[Image: test.png]

Now the histogram is an important feature, in the above example it is telling you the latency is less than or equal to 50 us since time of measurement (2017 measurements so far). You then look at the statistics and see the latency so far is spread across 1 us to 8 us, and over 2017 measurements, the average is 2.532 us.

Now if you have a problematic hardware somewhere, you'd see spikes (in the other rows under the histogram. Each row is called a bin). And you can identify the interval of the spikes by looking at the time history box (green box). That box will tell you if the spike is regular, or random.



Timer Frequency
It's actually the other way round, lower is better.

Code:

100 Hz is a typical choice for servers, SMP and NUMA systems
with lots of processors that may show reduced performance if
too many timer interrupts are occurring.
---
250 Hz is a good compromise choice allowing server performance
while also showing good interactive responsiveness even
on SMP and NUMA systems. If you are going to be using NTSC video
or multimedia, selected 300Hz instead.
---
300 Hz is a good compromise choice allowing server performance
while also showing good interactive responsiveness even
on SMP and NUMA systems and exactly dividing by both PAL and
NTSC frame rates for video and multimedia work.
---
1000 Hz is the preferred choice for desktop systems and other
systems requiring fast interactive responses to events.

Long story short, 1000 Hz kernel is really needed when you are using the computer directly. e.g. if you are using a keyboard/mouse, using a 1000 Hz kernel will make the system appear responsive.

If you are using a kernel that's on a 100 Hz timer... If you are a fast typer, or when you use a mouse, you may notice a degree of lag on the screen. E.g. keys show up slower, mouse don't quite move the way you want it to be.

The lag is not going to be an issue in Snakeoil because it is designed to run as headless. Graphical envrionments are VNC'ed in, the computer only designed to play audio and run server like applications. As such there is no user interaction, and it doesn't need that high frequency. Why? Because audio is buffered.

So what is this 100/250/300/1000 Hz anyway? It's the number of the times the OS will 'ask around' and see if any thing require attention. e.g. with keyboards, the OS will poll the keyboard driver 1000 times a second to see if a key is pressed. If you type at over 100 words per minute, that's like 500 too 800 characters a minute, and that's can be more than 5-10 keys a second. Believe it or not, if you use a kernel that's 100 Hz, the keys will appear on the screen slower (time taken to read keyboard input, time taken to render it on screen, etc).

For audio playback, audio is buffered (placed in memory), e.g. MPD has half a second of audio buffer, and the operating system will check this half second buffer 300 times a second to see if there's any data ready to be sent to the hardware. Checking this 1000 times a second if it's ready to be sent is not going to be an improvement. If you are using a player with absolutely no buffer, then a 1000 Hz kernel is necessary.

There's also this weird issue. If your audio system has a ground loop, using a 1000 Hz kernel actually generates some weird high frequency white noise..

And that timer is just one part of the story. Snakeoil is running a Real Time kernel. That means everything is preemptable. If you set your players to run at real time, that is going to cause unnecessary issues.

Question is if tickless is better than 100 Hz? I actually did this experiment a while back. Unfortunately I cannot remember the results now as I spent too much time on coding up Snakeoil OS, and not enough time listening..

[frednork]

@frednork ,

two quick comments.

Cyclic Test
What's your cyclictest latency numbers? Bear in mind Snakeoil uses nanoseconds (almost everybody else uses microseconds). So while it seem like your values look skyhigh, you need to divide by 1000. The reason to use nanoseconds is it's not difficult to get the latency down to 1 us with the right hardware when using Snakeoil OS. I did this with my Core2Duo a while back, and I still prefer my atom board, despite the worse latency numbers. For all intents and purposes 1 us latency vs 100 us isn't much in my point of view.

More information here. Also not sure if it's mentioned in this thread or somewhere else, the way cyclictest is running at the moment can be a bit deceptive. It is not showing you everything.

cyclicscope is a better presentation and more informative. This is something I intend to work on and hopefully get it out soon. In case you are unaware what that is, it looks something like this:
[Image: test.png]

Now the histogram is an important feature, in the above example it is telling you the latency is less than or equal to 50 us since time of measurement (2017 measurements so far). You then look at the statistics and see the latency so far is spread across 1 us to 8 us, and over 2017 measurements, the average is 2.532 us.

Now if you have a problematic hardware somewhere, you'd see spikes (in the other rows under the histogram. Each row is called a bin). And you can identify the interval of the spikes by looking at the time history box (green box). That box will tell you if the spike is regular, or random.



Timer Frequency
It's actually the other way round, lower is better.

Code:

100 Hz is a typical choice for servers, SMP and NUMA systems
with lots of processors that may show reduced performance if
too many timer interrupts are occurring.
---
250 Hz is a good compromise choice allowing server performance
while also showing good interactive responsiveness even
on SMP and NUMA systems. If you are going to be using NTSC video
or multimedia, selected 300Hz instead.
---
300 Hz is a good compromise choice allowing server performance
while also showing good interactive responsiveness even
on SMP and NUMA systems and exactly dividing by both PAL and
NTSC frame rates for video and multimedia work.
---
1000 Hz is the preferred choice for desktop systems and other
systems requiring fast interactive responses to events.

Long story short, 1000 Hz kernel is really needed when you are using the computer directly. e.g. if you are using a keyboard/mouse, using a 1000 Hz kernel will make the system appear responsive.

If you are using a kernel that's on a 100 Hz timer... If you are a fast typer, or when you use a mouse, you may notice a degree of lag on the screen. E.g. keys show up slower, mouse don't quite move the way you want it to be.

The lag is not going to be an issue in Snakeoil because it is designed to run as headless. Graphical envrionments are VNC'ed in, the computer only designed to play audio and run server like applications. As such there is no user interaction, and it doesn't need that high frequency. Why? Because audio is buffered.

So what is this 100/250/300/1000 Hz anyway? It's the number of the times the OS will 'ask around' and see if any thing require attention. e.g. with keyboards, the OS will poll the keyboard driver 1000 times a second to see if a key is pressed. If you type at over 100 words per minute, that's like 500 too 800 characters a minute, and that's can be more than 5-10 keys a second. Believe it or not, if you use a kernel that's 100 Hz, the keys will appear on the screen slower (time taken to read keyboard input, time taken to render it on screen, etc).

For audio playback, audio is buffered (placed in memory), e.g. MPD has half a second of audio buffer, and the operating system will check this half second buffer 300 times a second to see if there's any data ready to be sent to the hardware. Checking this 1000 times a second if it's ready to be sent is not going to be an improvement. If you are using a player with absolutely no buffer, then a 1000 Hz kernel is necessary.

There's also this weird issue. If your audio system has a ground loop, using a 1000 Hz kernel actually generates some weird high frequency white noise..

And that timer is just one part of the story. Snakeoil is running a Real Time kernel. That means everything is preemptable. If you set your players to run at real time, that is going to cause unnecessary issues.

Question is if tickless is better than 100 Hz? I actually did this experiment a while back. Unfortunately I cannot remember the results now as I spent too much time on coding up Snakeoil OS, and not enough time listening..

 

---

My cyclic test comes out around the 32000 mark give  or take, it doesnt seem that anything I have done since I started fiddling with snakeoil has made much difference, although not 100% sure on that.

I think I can probably make an incremental benefit if I can separate the roon service from the rest of the processes but still not clear how this may do this, If I startup with a roon service and then try to separate tasks to other cores snakeoil doesnt seem to "see" roon as a task.

Cyclicsope looks interesting, look forward to trying it. 

the 100hz idea also sounds interesting, is there a precedent or are you  lone wolf on this, 

There seem to be multiple strategies and these may be split up further depending on whether you are running a server/renderer box or separate boxes, low Hz, low power(arm or detuned more powerful processors) . I note the pi release is live so look forward to giving it a go but just trying to figure out what the best hardware might be to try, assuming not pi but that will probably be first cab off the rank.

Again thanks and appreciate your hard work!

---

Another question is what is your preferred server/player. Clearly not Roon. would like to try it

[agent_kith]

My cyclic test comes out around the 32000 mark give  or take, it doesnt seem that anything I have done since I started fiddling with snakeoil has made much difference, although not 100% sure on that.

I don't think it's going to do much difference regardless of the meddling. In theory changing the timer to pit/hpet/tsc etc will affect this. As is overclocking/underclockingi, or faster RAM. This test is mostly down to hardware speed.

Anyway your mark gives about 32 us (microseconds) which is relatively good. Not the best, but it's relatively responsive enough. I think my atom also yields similar results.

Using this as an example, that single number alone is not informative enough. What one should be interested to know is looking at the average of say 30 us, with a min of 1 us, and a max of 500 us. How many times did the computer's latency fall between > 30 us and < 500 us? Is there a pattern to it?

For audio playback, my theory is any latency less than 100 us is ok. In Snakeoil's cyclic test, that will show up as 100000 (because Snakeoil uses nanoseconds). Why is this enough? 100 us is actually 0.0001 seconds. Now when you run your applications in real time, and your computer can respond to the code in 0.0001 seconds, I reckon it's OK.
 

I think I can probably make an incremental benefit if I can separate the roon service from the rest of the processes but still not clear how this may do this, If I startup with a roon service and then try to separate tasks to other cores snakeoil doesnt seem to "see" roon as a task.

I cannot explain this but yes separation of tasks into cores does appear to work. I am thinking Roon takes too long to start, and my script to isolate the processes is run before that. .. Will have to add more code to check for that. This will hopefully be fixed in Blind Testing update 1.
 

Cyclicsope looks interesting, look forward to trying it. 

Me too. You can actually run this now from VNC. But my aim is to add this into the web app.
 

the 100hz idea also sounds interesting, is there a precedent or are you  lone wolf on this, 

I would hope not. Anybody who has a passing grade in Computer Operating Systems would know this.

There are many meanings to latencies. One of the more common one is lag. as an example, when you type a key on the keyboard, and the time it takes when that happen till the character appears on screen is called a lag. For this use case, using a 1000 Hz kernel will be better than a 100 Hz one. For just audio you don't need this.

So when do you need 1000 Hz kernels? When user interaction is important, or for syncing. Example:

1. When playing a movie, where video and audio has to sync up (lip sync)
   
2. Playing a game, when you fire a gun, the audio need to sound immediately
   
3. specific applications like voice over IP (VoIP), music mixing, video editing, etc
   

Long story short, if you need to 'line up multiple' inputs, then a higher timing is desirable.

Having said that, that's the theory. Does not necessarily mean 1000 Hz is bad. In some cases using something worse in theory actually yields better results. Over the next few days I'd build a 1000 Hz kernel, and you can try it and let me know how it goes. As always I will not recommend people what settings to use, it is important for people to find 'their own way' and tune Snakeoil to what they like, then share their results.

 

There seem to be multiple strategies and these may be split up further depending on whether you are running a server/renderer box or separate boxes, low Hz, low power(arm or detuned more powerful processors) . I note the pi release is live so look forward to giving it a go but just trying to figure out what the best hardware might be to try, assuming not pi but that will probably be first cab off the rank.

Personally I like to stay on a platform for years, study it and tweak the hell out of it. That focus is important because it means I am always clear when there is an improvement. Hence unlike a lot of people I havn't churn my audio gear. I am still using the same pre/amp and speakers for the past 10 years, the same DAC for the past 2 to 3. Using the same system for long periods gives me a stable reference. Unfortunately it also means I'm slow to adapt to big changes.

But I cannot force everybody to be on my pace. Pi is just the start, I think I will work on expressiobin in the next year. To be honest the software is well designed and will port to almost any architecture that has a debian OS. As requests come in, I will fill them in, but for myself I will still stick to my Atom.

Again thanks and appreciate your hard work!

My pleasure!
 

Another question is what is your preferred server/player. Clearly not Roon. would like to try it   

My system is simple. So I prefer LMS/Squeezelite. Music are WAV files and served over NAS (via CIFS).

If you want the best quality, I think MPD v20 so far is probably the best. Unfortunately that player is too good and it doesn't ease me into the music (too revealing?).. Sometimes I can feel my energy drained after a long MPD session. Squeezelite on the other hand, makes me feels recharged.

As for Roon, I think there's something wrong with it? See this thread.

[frednork]

Cyclicsope looks interesting, look forward to trying it. 

Me too. You can actually run this now from VNC. But my aim is to add this into the web app.

So will putty work or does it have to be vnc. I seem to be having a lot of trouble with vnc. I think its a network issue. 

Having said that, that's the theory. Does not necessarily mean 1000 Hz is bad. In some cases using something worse in theory actually yields better results. Over the next few days I'd build a 1000 Hz kernel, and you can try it and let me know how it goes. As always I will not recommend people what settings to use, it is important for people to find 'their own way' and tune Snakeoil to what they like, then share their results.

Look forward to trying that

If you want the best quality, I think MPD v20 so far is probably the best. Unfortunately that player is too good and it doesn't ease me into the music (too revealing?).. Sometimes I can feel my energy drained after a long MPD session. Squeezelite on the other hand, makes me feels recharged.

I notice there are multiple squeezlites , is it just feature differences or sq also. Is there a comparison thread somewhere

As for Roon, I think there's something wrong with it? See this thread

Ok, I am pretty sure a.dent still uses Roon with HQ player but will follow up

[agent_kith]

So will putty work or does it have to be vnc. I seem to be having a lot of trouble with vnc. I think its a network issue. 

It can be done via VNC if you want the graphical environment. Or better yet as a commandline, with the output to be loaded later.

 

Look forward to trying that

Hopefully will build that in the coming days. For 1000 Hz kernels to work well, you really need a computer with a very fast CPU. Let's say a computer can execute X instructions every second. A 100 Hz timer means a timeslice of X/100. And a 1000 Hz timer a timeslice of X/1000.

Definition of time slice is:

Code:

The period of time for which a process is allowed to run in a preemptive multitasking system is generally called the time slice or quantum. The scheduler is run once every time slice to choose the next process to run. The length of each time slice can be critical to balancing system performance vs process responsiveness - if the time slice is too short then the scheduler will consume too much processing time, but if the time slice is too long, processes will take longer to respond to input.

An interrupt is scheduled to allow the operating system kernel to switch between processes when their time slices expire, effectively allowing the processor’s time to be shared between a number of tasks, giving the illusion that it is dealing with these tasks in parallel (simultaneously). The operating system which controls such a design is called a multi-tasking system.

So when you are using a slow processor like an atom, running a high timer is going to be detrimental in theory. Basically the more process you set to execute in RT, potentially the worse the latency jitter will be.

This will only be evident when the software is poorly written. I don't think Squeezelite or MPD will fare worse, but can't vouch for the other players.

 

I notice there are multiple squeezlites , is it just feature differences or sq also. Is there a comparison thread somewhere

Not really.

Multiple versions is just that, multiple versions. e.g. if you have been listening to a version of Squeezelite for a long time, you may not want to change to the latest version. This is just a technique to reduce the number of variables when auditioning music. In my case, Squeezelite 1.6.4 is my 'control'. When testing new updates to Squeezelite, it's audio qualities is judged against the control.

Same with MPD. The control is v17, and all other versions of MPD is judged against it.

Frankly speaking though, you need to be very familiar with the track, and know what cues to listen to for a particular track. Without these cues all these different player versions will sound the same after a while.

Unless you're really serious about critical listening, don't think too much into it and just use the latest version, or the version that gives you the best features.


 

Ok, I am pretty sure a.dent still uses Roon with HQ player but will follow up