As eru said elsewhere, you've created a stream cipher. To be secure, the generated keystream (the thing you xor against the plaintext) must not have any biases: the attacker shouldn't be able to guess the contents of any keystream byte (with P > 1/2^8). Otherwise, an attacker can probabilistically recover parts of the plaintext.

I played around with your C implementation a little[1]. I put it into 8-bit mode, and generated 16 random keys and output their keystreams[2]. Here's the first 32 bytes of those keystreams, in hex:

0002010203fd050607b0090a0b250d0e0f8b111213a4151617f5191a1be6003e 3b4a00790369050607004b170b820d0e0fb7111213d0005f1796191a1b321d1e 00130102036b05060734090a0b260d0e0fca111264141516178700671b534ae8 df7c806903d10506077fcc670b583600dec617580a2b8b161712191a1b1c00a7 00260102032f1d0082a0090a0bd30d0e0f4ca8c616aa151617a400a34f1c9e1e af7cb175036705060786090a0bc90d0e0f2a111213831516173700501bfa1d1e 000236ca58110506079c00090b860d0e0f8f111251009e1617b4191a1b311d1e 00250102039700ec0729090a0b2a0d0e0f961112133a151617e3a200956764b8 002a0102031e050607ed003c0b260d0e0fcf111213c000151a18191a1bb21d1e 413c0102031505060745090a0b060d0e0fc6111213b815161784191a1b341d1e 00890102039500b807a200570ba70d0e0fa31112134c1516172d191a1ba03900 00cd01020332050607e100b10bdd0d0e0f0d111213261516170e191a1b041d1e 0076010203c30506074e03c7880069000eb8078d54134604a61a142d1a0600c3 00f101020344050607a8090a0b0e0d0e0f11111213e6151617f3191a1b291d1e 00f1007a0369aab10068090a0b53b1ac0fd1111213531516171800ee1bde1d1e 0036010203570506076c00d60b050d0e0f2f27583f03591617bc09471b141d1e

As you can see, there are some significant biases here. For example, as an attacker, once I have a ciphertext, I can guess that bytes 18 and 19 of the keystream were (hex) "1112", and have a very good chance of being right.

While the 32-bit version isn't as bad, I think there's still significant biases. I generated 24495 keys. Here's the distribution for keystream byte 1 (script at [3]):

   92    82    82   104    99    90    99   111    90   102    91    95    94    91    73   102
  109   106    97    99    88    99    90    96    88    97   101   100   108    76    87    87
   91    94   111    90    92   104    88    97    94   100    89   102    90    91    92    89
   98    96    89    94   111   111   105    90    87    89    93   104   100   110   109    93
   77   107   103    84    88    96    89    87    77    96    90    84    87   106   101    98
   99    99   114   102   104   106    95    91    95    92    94   104    95    88    93    91
   91    78   102    89   104    88    94   100   102   105    94   102   100   105    99    94
   87    89    86    93    95    77    82    83    99    94    88   106   106   101   101    91
   82    88    98   111   104    93   102    91    87    93   106    89   102    78    88   105
   91    93   105    84   101   100    94    93    94   107    88    86   114    84   112    98
   97    84   111    87    91    93    89    95    92    96    78    85    90   104    84    80
  103    95    98   114    90    89    91   110    89   100    87   107    95   109    83   103
  112   102    93    93    87    90   101    91   108   108    90   107   103    95   111   126
   88    97    74   111    97    99    95    95   102    97   122    94   106    94    97   103
   86    90   102    91    95   108    83    97    91   102    99    90   103   111    93    84
   87   107    91    96    77    98    96    85   108   110   116   100    95    89    72   100

I should really get to sleep so I didn't do the statistics on this, but I'm fairly sure that's not a uniform distrubtion (i.e., bytes aren't being drawn uniformly from [0:255]). Also, keystream byte 0 only ever has 128 values, instead of 255.

I did not look at 64-bit.

All in all, I wouldn't use your cryptosystem :) but there's no shame in that! Crypto is hard! and youre only going to get better.