***8217 ***8221

[firemonkey]

Why are they appearing in posts?

[firemonkey]

Just a laptop.

[Schmosby]

People posting with smart quotes turned on in iOS are causing the issue. I've written a fix, just waiting for site admin to apply it.

Are you using a macbook?

[Indigo_]

^ I will be passing your comments onto our administrator, who may be able to fix the issue himself

[Ronnie_Pickering]

It's because ..8217 and 8217, that wi8221 solve it

[Consolida]

This is happening all the time for me. I keep having to edit my posts or they are even more illegible than usual. It�s becoming very tedious

[Schmosby]

Turn off smart punctuation
Settings>General>Keyboard>Smart Punctuation

[limey123]

^ How do you fix this in Win 10?

[Occultus]

I've made a couple of changes, to see if this issue can be fixed. I suspect I may need to do a couple more things, so if there are any more issues, can you please let me know.

If it does still keep happening for you, it would be helpful if you could send me some details about what device you are using, what you are trying to post, etc., so I can try and replicate the problem at my end.

Please feel free to drop me a PM (or any of the mods) with the details if you encounter any more issues with it

[Schmosby]

Test:
“It is absolutely central that parliament should be able to call to account people who represent them as ministers, and at the moment we’re being told by a particular figure, who’s proud of it, that he’s more or less running the show,”

▓▓▓▓
▒▒▒▓▓
▒▒▒▒▒▓
▒▒▒▒▒▒▓
▒▒▒▒▒▒▓
▒▒▒▒▒▒▒▓
▒▒▒▒▒▒▒▓▓▓
▒▓▓▓▓▓▓░░░▓
▒▓░░░░▓░░░░▓
▓░░░░░░▓░▓░▓
▓░░░░░░▓░░░▓
▓░░▓░░░▓▓▓▓
▒▓░░░░▓▒▒▒▒▓
▒▒▓▓▓▓▒▒▒▒▒▓
▒▒▒▒▒▒▒▒▓▓▓▓
▒▒▒▒▒▓▓▓▒▒▒▒▓
▒▒▒▒▓▒▒▒▒▒▒▒▒▓
▒▒▒▓▒▒▒▒▒▒▒▒▒▓
▒▒▓▒▒▒▒▒▒▒▒▒▒▒▓
▒▓▒▓▒▒▒▒▒▒▒▒▒▓
▒▓▒▓▓▓▓▓▓▓▓▓▓
▒▓▒▒▒▒▒▒▒▓
▒▒▓▒▒▒▒▒▓

[Sisyphus]

Testing testing.

128 €
20ac ‚
201a ƒ
0192 „
201e …
2026 †
2020 ‡
2021 ˆ
02c6 ‰
2030 š
0160 ‹
2039 œ
0152 ž
144 ‘
2018 ’
2019 “
201c ”
201d •
2022 –
2013 —
2014 ˜
02dc ™
2122 š
0161 ›
203a œ
0153 ž
017e ÿ

[Sisyphus]

Now try editing.

128 €
20ac ‚
201a ƒ
0192 „
201e …
2026 †
2020 ‡
2021 ˆ
02c6 ‰
2030 š
0160 ‹
2039 œ
0152 ž
144 ‘
2018 ’
2019 “
201c ”
201d •
2022 –
2013 —
2014 ˜
02dc ™
2122 š
0161 ›
203a œ
0153 ž
017e ÿ

ETA: just because.

[Sisyphus]

Yay, we have Unicode back.

≠ ≡ ≢ ≣ ≤ ≥ ≦ ≧ ≨ ≩ ≪ ≫ ≬ ≭ ≮ ≯ ≰ ≱ ≲ ≳ ≴ ≵ ≶ ≷ ≸ ≹ ≺ ≻ ≼ ≽ ≾ ≿ ⊀ ⊁ ⊂ ⊃ ⊄ ⊅ ⊆ ⊇ ⊈ ⊉ ⊊ ⊋ ⊌ ⊍ ⊎ ⊏

ETA: And we have changed to UTF-8
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

[Sisyphus]

The punctuation that did get through to the database before now doesn't render. I guess it is not an issue though.

^^^^ That is probably why Unicode was blocked in the first place.

[Schmosby]

Yeah it's odd its not rendering, they only needed to replace *** with &# I would have thought.

[Schmosby]

ohhhhh &# is still in the swear filter.

[Sisyphus]

^^ I think that the fact that they don't render is how it should be if the database is ISO-8859-1. They are not printable characters in that encoding.

They shouldn't have made it in in the first place and should have been turned to Unicode and then redacted.

They are held natively so there is no *** to replace.

Also just doing a replace on *** might make more of a mess than expected. The swear filter is probably lossy and there will be no way to tell if the *** is there to replace the word ***.

Maybe the punctuation that did get through could be converted back if it still exists natively in the database.

The fact that the swear filter has not been changed does hint at a change of coding to UTF-8 in the database? The database could be holding the Unicode natively so it doesn't have to use the naughty prefix. That may explain why I am seeing the same unprintable character for different punctuation. I will double check what I am getting.

This solution to the issue makes it even more unclear why the prefix is being redacted if was not to simply disable Unicode.

[Schmosby]

No, they are held in the database with the stars.

It would be quite easy to replace using regex.

I'll test the changes on my copy and see what happens.

[Sisyphus]

Thanks Occultus for all your hard work behind the scenes.

[Sisyphus]

^^ Not the few extended punctuation characters that were getting trashed after editing. They must have been getting through natively.

See my post:-

Originally Posted by Sisyphus

The reson I was think what I did was that ISO-8859-1 doesn't contain the extended punctuation and yet they are still stored without being converted to Unicode. It is windows-1252 that does contain them.

If the board strictly used ISO-8859-1 throughout � � � � � � � � would never get to the database and yet here they are in the database.

As I said, I could be wrong.

[Schmosby]

Oh maybe when using Go Advanced. Ђ

[Sisyphus]

Do you think the database has been changed to UTF-8?

How else could all this Unicode bypass the swear filter?

[Schmosby]

I just changed my database to UTF-8 and the forum and I get the same bad characters.

It's still stored correctly in the database.

[Schmosby]

OK, I've found the issue. It needs changing in a config file too.

I'll message him.

[Sisyphus]

I don’t think the loss of the former punctuation marks is due to settings. My guess would be that the database has been converted to UTF-8 from ISO-8859-1 rather than from windows-1252.

The extended punctuation marks that formerly got into the database were windows-1252. Whatever software transcoded the database didn’t know what to do with the unprintable (in ISO-8859-1) characters and replaced them with Unicode FFFD.

Under HTML5 I think the transcoder ‘should’ have assumed windows-1252 for anything it didn’t recognize and if it had it would have got it right.

This makes me think there may be a mix of HTML and HTML5 compatible code in the engine room and may be how the extended punctuation got into the database in the first place.

If this is what has happened then some information may have been lost as all of the extended punctuation is now FFFD. Some effort could be made to mitigate this but I guess it will soon fade into the past and can sit alongside the ***8221; as a curiosity.

[Schmosby]

No I tested it on my install, changing the setting made it display correctly.

[Sisyphus]

Then I am missing something.

What setting did you change? What does it do?

[Sisyphus]

Unless it is transcoding on the fly then it can't be corrected by changng a setting now.

If it is transcoding on the fly then that implies the database is still in the old format.

If that is the case then the database cannot hold Unicode because the swear filter will stomp on it.



To see what I am seeing I think the database must now be UTF-8 and that must have happened at one epoch. That being the case then the information has now been lost.

But I have been wrong before.

[Schmosby]

I think the setting changes the way mysqli handles the data.

    //    ****** MySQLI OPTIONS ***** 
    // When using MySQL 4.1+, MySQLi should be used to connect to the database.
    // If you need to set the default connection charset because your database
    // is using a charset other than latin1, you can set the charset here.
    // If you don't set the charset to be the same as your database, you
    // may receive collation errors.  Ignore this setting unless you
    // are sure you need to use it. 
// $config['Mysqli']['charset'] = 'utf8';

The data is stored correctly in the database, it's only on the site it shows as question marks. Once the setting is changed the unicide will show correctly.

[Sisyphus]

Something still doesn't ring true to me. My brain will gnaw on this when it could be spending time not gnawing on it.