rjwilmsi

My take wasn't that they are claiming SOTA, just that they are claiming same/better accuracy than whisper "tiny" and "base" models at higher performance versus original whisper.

The WER scores on the larger whisper models are lower (better) than "tiny"/"base", so none of these small models are close to SOTA either way.

I've played quite a lot with all the whisper models up to "medium" size. Mostly via faster-whisper as original OpenAI whisper only seem to care for/optimize performance for GPU.

I would agree that the "tiny" model has a clear drop off in accuracy, not good enough for anything real (even if transcribing your own speech, the error rate means too much editing needed). In my experience, accuracy can be more of a problem on shorter sentences because there is less context to help it.

I think for serious use (on GPU) it would be the "medium" or "large" models only. There is now a "large-turbo" model which is apparently faster than "medium" (on GPU) but more accurate than "medium" - haven't tried it yet.

On CPU for personal use (faster-whisper, CPU) I have found "base" is usable, "small" is good. On a laptop CPU though "small" is slow for real time. "Medium" is more accurate, though mostly just on punctuation, far too slow for CPU. Of course all models will get some uncommon surnames, place names wrong.

Since OpenAI have re-released the "large" models twice and now done a "large-turbo" I hope that they will re-release the smaller models too so that the smallest models become more useful.

These moonshine models are compared to original OpenAI whisper, but really I'd say they need to compare to faster-whisper: multiple projects are faster than original OpenAI whisper.

Yes, I'm aware of the Agile Octopus tariff etc. but answered based on what >95% of UK retail consumers are doing.

From historical data, Octopus would also pass on a negative spot price to customers on that tariff so consumers could sometimes get paid to use electricity.

As you say I'd imagine that certain customers with high flexibility, maybe/particularly those with some battery storage (or an EV to charge), could make huge savings.

I don't know what sort of margin Octopus apply on that tariff, I expect it's a bit higher than the 3 p / kWh you suggest (when normal tariffs would average something like 15 p / kWh difference between average wholesale and retail).

You are mixing the grid and retail electricity prices.

The retail electricity prices are linked to the wholesale gas cost. So consumers pay the same unit rate for electricity regardless of source/time of day etc. (simplification). Average 23p / kWh or so.

However, the producers of the electricity get paid the grid spot price. Plus any source-specific government-agreed subsidies (for nuclear, or the strike price for wind auctions).

So the typical/average UK grid electricity spot price is around £70 per MWh or 7p per kWh, so while retail is 23p / kWh a solar producer doesn't get paid 23p.

What you will also find is that the spot price, while it varies a lot, tends to be higher in the UK late afternoon / early evening, and higher in the winter than the summer, both of which are normally times when solar output is lower.

I take the opposite view - as more people have electric cars there are more batteries available. With some sort of dynamic pricing, and bearing in mind most commuter mileage needs only 1 full charge a week, you could encourage consumers to charge up when it's windy and/or hold off charging when it's not, so better matching demand to supply and reducing curtailment.

To me it made sense to price electricity at retail based on gas price when gas was the main source and the cost for that source is mostly about the market fuel cost. (There were and are other price guarantee schemes for nuclear where the ongoing fuel cost is only a small part of total costs. And the grid electricity price for wind is agreed by another set of rules - strike price).

However, we're now beyond that point so it seems to me that the pricing model should change. With the planned growth rate of new wind (and solar) by 2030 for the UK I assume it's going to happen one way or another.

23.4 p/kWh sounds about right. When prices go up next month (October 2024) I will be paying 30.5p / kWh day and 13.2p / kWh overnight (midnight to 7 am). Those are retail/consumer prices subject to government price cap, I believe commercial customers pay more.

Agree. It's about whether a buyer would reasonably expect the price to be correct.

UK law for consumer goods and sales, I would say generally doesn't require the buyer to be an expert. Laptops are items that are frequently included in sales and clearance sales at 50% off or more do happen. £400 is still a normal price to pay for a laptop. So I'd say the buyer could reasonably expect a laptop discounted to £400 to be the correct price, therefore HP were unreasonable to try to blame the buyer.

Of course. Small Claims process in the UK has a fee of about 10% of claim value and covers claims up to £10,000. If you win the other party has to repay those fees. No legal fees allowed. So would be the right process here - presumably as HP are in breach of contract the laptop buyer would win a case for a claim of the extra cost of the same laptop at full price.

That's the implication. If the distil models are same format as original openai models then the Distil models can be converted for faster-whisper use as per the conversion instructions on https://github.com/guillaumekln/faster-whisper/

So then we'll see whether we get the 6x model speedup on top of the stated 4x faster-whisper code speedup, at same/nearly same accuracy.

I would generally start with the assumption that if something is significantly faster the accuracy has to suffer a bit, but increasing model size and/or settings such as beam size to compensate should allow same accuracy and higher performance (just not all of the stated performance gain).

Yes, it's not that clear to me either what test sets get a 10% error rate. Because in my use (native English dictation or native English podcast transcription) the small or medium original whisper models have what I'll call a "discrepancy" rate of say 1-2% which is mostly punctuation and "umms/errs" inclusion or not. The actual "error" rate is below 1% in my experience, and excluding surnames, brands and place names that I don't know how to spell either the remaining errors tend to be minor (missed plural etc.).

So I infer that these data sets are some deliberately difficult audio: call centre recordings with lots of background noise, phoneline quality audio etc. Maybe non-native speakers. If I only heard that sort of audio once I also might have an error rate of 10%.

I agree. When using the small or medium en models either for real-time speech recognition of a native English speaker or for transcribing podcasts of native English speakers the error rate is nowhere near 10%. I might say it's something like 1% of which the majority of errors are possibly subjective decisions about punctuation. But I have found the error rates are much higher on the tiny model and higher on the base model.

I assume therefore that the 10% word error rate is on very difficult audio such as pilots speaking to Air Traffic Control (distorted or clipped microphones with significant background noise), which I personally find can be difficult to 100% understand even though I'm a native English speaker and when both pilots and air traffic control are native English speakers.

I understand that, though I think significant speedups can be useful at multiple levels. So for me for example I am using either the base or small model on beam size of 1 with faster-whisper for real-time dictation on a laptop CPU (Rzyen 4500U). The recognition time is just that bit too high when using a larger beam size or is much too high when using the medium model. So if these models offer a decent speed up it means I can either increase beam size or go up a model size which will lead to good improvement in accuracy I think - With real-time dictation I find that small errors are quite annoying to deal with so any improvement in accuracy is really useful.

At a larger level, say an exercise to transcribe a back catalogue of audio might need a $1000 GPU with the current model speeds to get the job done in a reasonable time. With models that run 6x faster it might be that a $200 GPU is sufficient. That could be quite a significant saving for a small company or charity etc.